==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-5497cb87d7-mlld4 Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-1b0769c8-7vmj/10.142.0.111 Start Time: Sat, 20 Apr 2024 21:31:32 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=5497cb87d7 skaffold.dev/run-id=c5947d2f-65a0-4ce7-9e6d-e48a6b68f4e6 Annotations: Status: Running IP: 10.106.42.193 IPs: IP: 10.106.42.193 Controlled By: ReplicaSet/lodemon-5497cb87d7 Containers: lodemon: Container ID: containerd://7dc0cfbf4b353ecc994741cd62e56f1d12eeab7ae3d8277040bf78f977d83873 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:03f76e6326d713f4d73dc474e0e25fbbbaecc7c1d8c0590bfd94d68ac0d98c97 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Sat, 20 Apr 2024 21:31:33 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-w4nsz (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-w4nsz: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 43m default-scheduler Successfully assigned xlou/lodemon-5497cb87d7-mlld4 to gke-xlou-cdm-default-pool-1b0769c8-7vmj Normal Pulling 43m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 43m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 233.834643ms (233.849056ms including waiting) Normal Created 43m kubelet Created container lodemon Normal Started 43m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.42.193:8080 Press CTRL+C to quit 22:32:30 INFO 22:32:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:30 INFO [loop_until]: OK (rc = 0) 22:32:30 DEBUG --- stdout --- 22:32:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:30 DEBUG --- stderr --- 22:32:30 DEBUG 22:32:30 INFO 22:32:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:30 INFO [loop_until]: OK (rc = 0) 22:32:30 DEBUG --- stdout --- 22:32:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:30 DEBUG --- stderr --- 22:32:30 DEBUG 22:32:30 INFO 22:32:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:30 INFO [loop_until]: OK (rc = 0) 22:32:30 DEBUG --- stdout --- 22:32:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:30 DEBUG --- stderr --- 22:32:30 DEBUG 22:32:30 INFO 22:32:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:30 INFO [loop_until]: OK (rc = 0) 22:32:30 DEBUG --- stdout --- 22:32:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:30 DEBUG --- stderr --- 22:32:30 DEBUG 22:32:30 INFO 22:32:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:30 INFO [loop_until]: OK (rc = 0) 22:32:30 DEBUG --- stdout --- 22:32:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:30 DEBUG --- stderr --- 22:32:30 DEBUG 22:32:30 INFO 22:32:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:30 INFO [loop_until]: OK (rc = 0) 22:32:30 DEBUG --- stdout --- 22:32:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:30 DEBUG --- stderr --- 22:32:30 DEBUG 22:32:30 INFO 22:32:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:30 INFO [loop_until]: OK (rc = 0) 22:32:30 DEBUG --- stdout --- 22:32:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:30 DEBUG --- stderr --- 22:32:30 DEBUG 22:32:31 INFO 22:32:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:31 INFO [loop_until]: OK (rc = 0) 22:32:31 DEBUG --- stdout --- 22:32:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:31 DEBUG --- stderr --- 22:32:31 DEBUG 22:32:31 INFO 22:32:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:31 INFO [loop_until]: OK (rc = 0) 22:32:31 DEBUG --- stdout --- 22:32:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:31 DEBUG --- stderr --- 22:32:31 DEBUG 22:32:31 INFO 22:32:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:31 INFO [loop_until]: OK (rc = 0) 22:32:31 DEBUG --- stdout --- 22:32:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:31 DEBUG --- stderr --- 22:32:31 DEBUG 22:32:31 INFO 22:32:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:31 INFO [loop_until]: OK (rc = 0) 22:32:31 DEBUG --- stdout --- 22:32:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:31 DEBUG --- stderr --- 22:32:31 DEBUG 22:32:31 INFO 22:32:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:31 INFO [loop_until]: OK (rc = 0) 22:32:31 DEBUG --- stdout --- 22:32:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:31 DEBUG --- stderr --- 22:32:31 DEBUG 22:32:31 INFO 22:32:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:31 INFO [loop_until]: OK (rc = 0) 22:32:31 DEBUG --- stdout --- 22:32:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:31 DEBUG --- stderr --- 22:32:31 DEBUG 22:32:31 INFO 22:32:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:31 INFO [loop_until]: OK (rc = 0) 22:32:31 DEBUG --- stdout --- 22:32:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:31 DEBUG --- stderr --- 22:32:31 DEBUG 22:32:31 INFO 22:32:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:32 INFO [loop_until]: OK (rc = 0) 22:32:32 DEBUG --- stdout --- 22:32:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:32 DEBUG --- stderr --- 22:32:32 DEBUG 22:32:32 INFO 22:32:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:32 INFO [loop_until]: OK (rc = 0) 22:32:32 DEBUG --- stdout --- 22:32:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:32 DEBUG --- stderr --- 22:32:32 DEBUG 22:32:32 INFO 22:32:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:32 INFO [loop_until]: OK (rc = 0) 22:32:32 DEBUG --- stdout --- 22:32:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:32 DEBUG --- stderr --- 22:32:32 DEBUG 22:32:32 INFO 22:32:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:32 INFO [loop_until]: OK (rc = 0) 22:32:32 DEBUG --- stdout --- 22:32:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:32 DEBUG --- stderr --- 22:32:32 DEBUG 22:32:32 INFO 22:32:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:32 INFO [loop_until]: OK (rc = 0) 22:32:32 DEBUG --- stdout --- 22:32:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:32 DEBUG --- stderr --- 22:32:32 DEBUG 22:32:32 INFO 22:32:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:32 INFO [loop_until]: OK (rc = 0) 22:32:32 DEBUG --- stdout --- 22:32:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:32 DEBUG --- stderr --- 22:32:32 DEBUG 22:32:32 INFO 22:32:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:32 INFO [loop_until]: OK (rc = 0) 22:32:32 DEBUG --- stdout --- 22:32:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:32 DEBUG --- stderr --- 22:32:32 DEBUG 22:32:32 INFO 22:32:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:33 INFO [loop_until]: OK (rc = 0) 22:32:33 DEBUG --- stdout --- 22:32:33 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:33 DEBUG --- stderr --- 22:32:33 DEBUG 22:32:33 INFO 22:32:33 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:33 INFO [loop_until]: OK (rc = 0) 22:32:33 DEBUG --- stdout --- 22:32:33 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:33 DEBUG --- stderr --- 22:32:33 DEBUG 22:32:33 INFO 22:32:33 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:33 INFO [loop_until]: OK (rc = 0) 22:32:33 DEBUG --- stdout --- 22:32:33 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:33 DEBUG --- stderr --- 22:32:33 DEBUG 22:32:33 INFO 22:32:33 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:33 INFO [loop_until]: OK (rc = 0) 22:32:33 DEBUG --- stdout --- 22:32:33 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:33 DEBUG --- stderr --- 22:32:33 DEBUG 22:32:33 INFO 22:32:33 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:33 INFO [loop_until]: OK (rc = 0) 22:32:33 DEBUG --- stdout --- 22:32:33 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:33 DEBUG --- stderr --- 22:32:33 DEBUG 22:32:33 INFO 22:32:33 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 22:32:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:33 INFO [loop_until]: OK (rc = 0) 22:32:33 DEBUG --- stdout --- 22:32:33 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 22:32:33 DEBUG --- stderr --- 22:32:33 DEBUG 22:32:33 INFO Initializing monitoring instance threads 22:32:33 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 22:32:33 INFO Starting instance threads 22:32:33 INFO 22:32:33 INFO Thread started 22:32:33 INFO [loop_until]: kubectl --namespace=xlou top node 22:32:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:33 INFO 22:32:33 INFO Thread started 22:32:33 INFO [loop_until]: kubectl --namespace=xlou top pods 22:32:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 INFO Thread started 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 INFO Thread started 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 INFO Thread started 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 INFO Thread started 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713648753" 22:32:33 INFO Thread started 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713648753" 22:32:33 INFO Thread started 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648753" 22:32:33 INFO Thread started 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648753" 22:32:33 INFO Thread started 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648753" 22:32:33 INFO Thread started 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 INFO Thread started 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 INFO Thread started 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 INFO Thread started 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 INFO Thread started 22:32:33 DEBUG --- status code --- 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG --- status code --- 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG --- status code --- 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 INFO Thread started 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG --- status code --- 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG --- http response --- 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 INFO Thread started 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:33 DEBUG --- http response --- 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:33 INFO Thread started 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:33 DEBUG ----- output ----- 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 INFO Thread started 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG { 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG { 22:32:33 DEBUG { 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 DEBUG { 22:32:33 INFO Thread started 22:32:33 DEBUG { 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG { 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG "status": "success", 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG { 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG "status": "success", 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG "status": "success", 22:32:33 INFO Thread started 22:32:33 DEBUG { 22:32:33 INFO [loop_until]: OK (rc = 0) 22:32:33 DEBUG { 22:32:33 DEBUG { 22:32:33 DEBUG { 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:32:33 DEBUG "data": { 22:32:33 DEBUG --- http response --- 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG "data": { 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG "data": { 22:32:33 DEBUG "data": { 22:32:33 DEBUG "data": { 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG "data": { 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 DEBUG "status": "success", 22:32:33 INFO Thread started 22:32:33 DEBUG --- stdout --- 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 INFO [loop_until]: OK (rc = 0) 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG "data": { 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG "data": { 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8706Mi am-767f6b67b5-qcqz2 7m 8822Mi am-767f6b67b5-vbltz 6m 8842Mi ds-cts-0 15m 5657Mi ds-cts-1 13m 12257Mi ds-cts-2 14m 11755Mi ds-idrepo-0 13m 3931Mi ds-idrepo-1 14m 3996Mi ds-idrepo-2 16m 3901Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1179Mi idm-b85f9d978-wnhxb 3m 1240Mi lodemon-5497cb87d7-mlld4 2m 54Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-967d957f7-zk66v 1m 2539Mi 22:32:33 INFO Thread started 22:32:33 DEBUG "data": { 22:32:33 DEBUG "data": { 22:32:33 DEBUG "data": { 22:32:33 DEBUG { 22:32:33 DEBUG "result": [ 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG --- stdout --- 22:32:33 DEBUG "result": [ 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG "result": [ 22:32:33 DEBUG "result": [ 22:32:33 DEBUG "result": [ 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG "result": [ 22:32:33 DEBUG "resultType": "vector", 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG --- stderr --- 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 DEBUG "resultType": "vector", 22:32:33 INFO Thread started 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG { 22:32:33 DEBUG { 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:33 DEBUG { 22:32:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6289Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 71m 0% 7673Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 11423Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3402Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 71m 0% 8893Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 164m 0% 5196Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 11719Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 81m 0% 3214Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 61m 0% 11075Mi 9% gke-xlou-cdm-ds-0f601751-j488 66m 0% 7292Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 72m 0% 17372Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 72m 0% 7950Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 70m 0% 14430Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 71m 0% 10477Mi 17% 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG "result": [ 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:33 DEBUG { 22:32:33 DEBUG { 22:32:33 DEBUG { 22:32:33 DEBUG { 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:33 DEBUG "result": [ 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG 22:32:33 DEBUG "result": [ 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 DEBUG "result": [ 22:32:33 DEBUG "result": [ 22:32:33 INFO Thread started 22:32:33 DEBUG "data": { 22:32:33 DEBUG "metric": { 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:33 DEBUG ----- output ----- 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG --- stderr --- 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG { 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG "metric": { 22:32:33 DEBUG "metric": { 22:32:33 DEBUG "metric": { 22:32:33 DEBUG "metric": { 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG { 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG { 22:32:33 DEBUG { 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG { 22:32:33 DEBUG "resultType": "vector", 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713648753" 22:32:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:32:33 INFO Thread started 22:32:33 DEBUG "data": { 22:32:33 DEBUG { 22:32:33 DEBUG "metric": { 22:32:33 DEBUG { 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG "metric": { 22:32:33 DEBUG { 22:32:33 DEBUG "pod": "ds-idrepo-1" 22:32:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:32:33 DEBUG "pod": "ds-idrepo-1" 22:32:33 DEBUG "pod": "ds-idrepo-1" 22:32:33 DEBUG { 22:32:33 DEBUG "metric": { 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG "metric": { 22:32:33 DEBUG "metric": { 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG "metric": { 22:32:33 DEBUG "result": [ 22:32:33 DEBUG }, 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG "resultType": "vector", 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648753" 22:32:33 DEBUG "status": "success", 22:32:33 INFO Thread started 22:32:33 DEBUG "pod": "ds-idrepo-1" 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG "instance": "10.142.0.107:9100" 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG }, 22:32:33 DEBUG }, 22:32:33 DEBUG }, 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG "pod": "am-767f6b67b5-9zw22" 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:33 DEBUG "instance": "10.142.0.107:9100" 22:32:33 DEBUG "pod": "am-767f6b67b5-9zw22" 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG "instance": "10.142.0.107:9100" 22:32:33 DEBUG { 22:32:33 DEBUG "value": [ 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG "result": [ 22:32:33 DEBUG "data": { 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648753" 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 INFO Thread started 22:32:33 DEBUG }, 22:32:33 DEBUG "data": { 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG }, 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:32:33 DEBUG "data": { 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG "value": [ 22:32:33 DEBUG }, 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "data": { 22:32:33 DEBUG }, 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG }, 22:32:33 DEBUG }, 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG }, 22:32:33 DEBUG "metric": { 22:32:33 DEBUG 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG { 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "resultType": "vector", 22:32:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648753" 22:32:33 DEBUG "value": [ 22:32:33 INFO Thread started 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:32:33 DEBUG ----- output ----- 22:32:33 ERROR [http_cmd]: ERROR 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "result": [ 22:32:33 DEBUG { 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "value": [ 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:33 DEBUG "pod": "ds-cts-0" 22:32:33 DEBUG 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG "metric": { 22:32:33 DEBUG "result": [ 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG 22:32:33 DEBUG "result": [ 22:32:33 DEBUG 22:32:33 INFO All threads has been started 22:32:33 DEBUG ----- output ----- 127.0.0.1 - - [20/Apr/2024 22:32:33] "GET /monitoring/start HTTP/1.1" 200 - 22:32:33 DEBUG { 22:32:33 ERROR --- status code --- 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG "metric": { 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG "result": [ 22:32:33 DEBUG { 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG }, 22:32:33 DEBUG 22:32:33 DEBUG "pod": "ds-idrepo-0" 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:32:33 DEBUG { 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG 22:32:33 DEBUG { 22:32:33 DEBUG 22:32:33 DEBUG { 22:32:33 DEBUG "status": "success", 22:32:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG "pod": "ds-idrepo-0" 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG { 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG { 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG "value": [ 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG [...] 22:32:33 DEBUG }, 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG "metric": { 22:32:33 DEBUG 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:32:33 DEBUG "metric": { 22:32:33 DEBUG 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG "data": { 22:32:33 ERROR --- http response --- 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG }, 22:32:33 DEBUG [...] 22:32:33 DEBUG [...] 22:32:33 DEBUG "metric": { 22:32:33 DEBUG "pod": "am-767f6b67b5-9zw22" 22:32:33 DEBUG "data": { 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG 22:32:33 DEBUG "value": [ 22:32:33 DEBUG { 22:32:33 DEBUG "pod": "ds-idrepo-0" 22:32:33 DEBUG [...] 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG "pod": "am-767f6b67b5-9zw22" 22:32:33 DEBUG [...] 22:32:33 DEBUG "data": { 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG "result": [ 22:32:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:32:33 DEBUG [...] 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG [...] 22:32:33 DEBUG }, 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG [...] 22:32:33 DEBUG [...] 22:32:33 INFO [http_cmd]: http status code OK 22:32:33 DEBUG [...] 22:32:33 DEBUG "data": { 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG }, 22:32:33 DEBUG 22:32:33 DEBUG { 22:32:33 DEBUG }, 22:32:33 DEBUG 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG { 22:32:33 DEBUG "metric": { 22:32:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "result": [ 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG --- status code --- 22:32:33 DEBUG 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG "data": { 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 22:32:33 DEBUG "result": [ 22:32:33 DEBUG [...] 22:32:33 DEBUG "pod": "ds-cts-0" 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG "pod": "overseer-0-967d957f7-zk66v" 22:32:33 DEBUG 22:32:33 DEBUG { 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:32:33 DEBUG 22:32:33 DEBUG "result": [ 22:32:33 DEBUG [...] 22:32:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:32:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:33 DEBUG 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG "data": { 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG { 22:32:33 DEBUG 22:32:33 DEBUG }, 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG "pod": "ds-cts-1" 22:32:33 DEBUG }, 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 22:32:33 DEBUG "metric": { 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG --- http response --- 22:32:33 DEBUG 22:32:33 DEBUG { 22:32:33 DEBUG 22:32:33 DEBUG }, 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG [...] 22:32:33 DEBUG "result": [ 22:32:33 DEBUG 22:32:33 DEBUG "pod": "ds-cts-1" 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG 22:32:33 DEBUG "instance": "10.142.0.43:9100" 22:32:33 DEBUG "metric": { 22:32:33 DEBUG 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:32:33 DEBUG [...] 22:32:33 DEBUG 22:32:33 DEBUG }, 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG 22:32:33 DEBUG "pod": "am-767f6b67b5-9zw22" 22:32:33 DEBUG "instance": "10.142.0.43:9100" 22:32:33 DEBUG "pod": "am-767f6b67b5-vbltz" 22:32:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:32:33 DEBUG "instance": "10.142.0.43:9100" 22:32:33 DEBUG "metric": { 22:32:33 DEBUG 22:32:33 DEBUG "value": [ 22:32:33 DEBUG { 22:32:33 DEBUG 22:32:33 DEBUG { 22:32:33 DEBUG 22:32:33 DEBUG }, 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 22:32:33 DEBUG }, 22:32:33 DEBUG "pod": "ds-cts-0" 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG }, 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "0" 22:32:33 DEBUG [...] 22:32:33 DEBUG }, 22:32:33 DEBUG }, 22:32:33 DEBUG }, 22:32:33 DEBUG ----- output ----- 22:32:33 DEBUG { 22:32:33 DEBUG "pod": "am-767f6b67b5-9zw22" 22:32:33 DEBUG 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG 22:32:33 DEBUG "metric": { 22:32:33 DEBUG [...] 22:32:33 DEBUG "result": [ 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG "0" 22:32:33 DEBUG "value": [ 22:32:33 DEBUG }, 22:32:33 DEBUG "pod": "overseer-0-967d957f7-zk66v" 22:32:33 DEBUG 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 22:32:33 DEBUG "pod": "am-767f6b67b5-vbltz" 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG ] 22:32:33 DEBUG 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "value": [ 22:32:33 DEBUG }, 22:32:33 DEBUG "status": "success", 22:32:33 DEBUG }, 22:32:33 DEBUG "pod": "ds-cts-2" 22:32:33 DEBUG "0.024439922738662967" 22:32:33 DEBUG "data": { 22:32:33 DEBUG 22:32:33 DEBUG "pod": "am-767f6b67b5-9zw22" 22:32:33 DEBUG 22:32:33 DEBUG { 22:32:33 DEBUG [...] 22:32:33 DEBUG ] 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG "value": [ 22:32:33 DEBUG }, 22:32:33 DEBUG 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG 22:32:33 DEBUG }, 22:32:33 DEBUG "0" 22:32:33 DEBUG } 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "data": { 22:32:33 DEBUG "value": [ 22:32:33 DEBUG }, 22:32:33 DEBUG ] 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG "pod": "ds-idrepo-2" 22:32:33 DEBUG }, 22:32:33 DEBUG 22:32:33 DEBUG "metric": { 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG "0.0006666666666660604" 22:32:33 DEBUG 22:32:33 DEBUG "value": [ 22:32:33 DEBUG [...] 22:32:33 DEBUG "2759004160" 22:32:33 DEBUG "pod": "ds-idrepo-2" 22:32:33 DEBUG "value": [ 22:32:33 DEBUG ] 22:32:33 DEBUG ] 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG "25440.766666666666" 22:32:33 DEBUG "0" 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG "resultType": "vector", 22:32:33 DEBUG 22:32:33 DEBUG "value": [ 22:32:33 DEBUG } 22:32:33 DEBUG "result": [ 22:32:33 DEBUG }, 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 22:32:33 DEBUG "pod": "ds-idrepo-0" 22:32:33 DEBUG 22:32:33 DEBUG ] 22:32:33 DEBUG ] 22:32:33 DEBUG 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG 22:32:33 DEBUG ] 22:32:33 DEBUG }, 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG "pod": "am-767f6b67b5-vbltz" 22:32:33 DEBUG 22:32:33 DEBUG ] 22:32:33 DEBUG ] 22:32:33 DEBUG "27576.433333333334" 22:32:33 DEBUG "result": [ 22:32:33 DEBUG 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG ] 22:32:33 DEBUG { 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 22:32:33 DEBUG "pod": "ds-idrepo-2" 22:32:33 DEBUG }, 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG 22:32:33 DEBUG "0" 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "0" 22:32:33 DEBUG ] 22:32:33 DEBUG } 22:32:33 DEBUG }, 22:32:33 DEBUG [...] 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG { 22:32:33 DEBUG 22:32:33 DEBUG "0" 22:32:33 DEBUG } 22:32:33 DEBUG "metric": { 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG 22:32:33 DEBUG }, 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "pod": "am-767f6b67b5-vbltz" 22:32:33 DEBUG } 22:32:33 DEBUG [...] 22:32:33 DEBUG ] 22:32:33 DEBUG 22:32:33 DEBUG ] 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG ] 22:32:33 DEBUG } 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 22:32:33 DEBUG ] 22:32:33 DEBUG ] 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG [...] 22:32:33 DEBUG ] 22:32:33 DEBUG } 22:32:33 DEBUG "instance": "10.142.0.107:9100" 22:32:33 DEBUG "0" 22:32:33 DEBUG 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG }, 22:32:33 DEBUG ] 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG "pod": "ds-cts-2" 22:32:33 DEBUG "0" 22:32:33 DEBUG ] 22:32:33 DEBUG } 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG "metric": { 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG }, 22:32:33 DEBUG ] 22:32:33 DEBUG [...] 22:32:33 DEBUG 22:32:33 DEBUG "0" 22:32:33 DEBUG "value": [ 22:32:33 DEBUG } 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG }, 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG } 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG "instance": "10.142.0.107:9100" 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG "value": [ 22:32:33 DEBUG } 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG ] 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG } 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "0" 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG "pod": "am-767f6b67b5-vbltz" 22:32:33 DEBUG }, 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG 22:32:33 DEBUG ] 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG "0" 22:32:33 DEBUG "pod": "ds-cts-2" 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG ] 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG }, 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG 22:32:33 DEBUG [...] 22:32:33 DEBUG ] 22:32:33 DEBUG ] 22:32:33 DEBUG }, 22:32:33 DEBUG "value": [ 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 22:32:33 DEBUG "pod": "am-767f6b67b5-vbltz" 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG "pod": "am-767f6b67b5-vbltz" 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG "1.3333333333333333" 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG 22:32:33 DEBUG }, 22:32:33 DEBUG [...] 22:32:33 DEBUG }, 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG ] 22:32:33 DEBUG "0" 22:32:33 DEBUG } 22:32:33 DEBUG "0" 22:32:33 DEBUG 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 22:32:33 DEBUG "value": [ 22:32:33 DEBUG 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG ] 22:32:33 DEBUG [...] 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG "pod": "ds-cts-2" 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG "1" 22:32:33 DEBUG 22:32:33 DEBUG "101.875" 22:32:33 DEBUG }, 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG 22:32:33 DEBUG ] 22:32:33 DEBUG "instance": "10.142.0.43:9100" 22:32:33 DEBUG ] 22:32:33 DEBUG "value": [ 22:32:33 DEBUG ] 22:32:33 DEBUG } 22:32:33 DEBUG 22:32:33 DEBUG 22:32:33 DEBUG }, 22:32:33 DEBUG } 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG "instance": "10.142.0.43:9100" 22:32:33 DEBUG "value": [ 22:32:33 DEBUG ] 22:32:33 DEBUG "0" 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG }, 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG } 22:32:33 DEBUG "value": [ 22:32:33 DEBUG "4131187715072" 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG 1713648753, 22:32:33 DEBUG ] 22:32:33 DEBUG ] 22:32:33 DEBUG "0.0009000000000014552" 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG ] 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG ] 22:32:33 DEBUG } 22:32:33 DEBUG } 22:32:33 DEBUG } 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713648813" 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713648813" 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648813" 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648813" 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 INFO 22:33:33 INFO [loop_until]: kubectl --namespace=xlou top pods 22:33:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 INFO 22:33:33 DEBUG --- status code --- 22:33:33 ERROR [http_cmd]: ERROR 22:33:33 ERROR --- status code --- 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG --- status code --- 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648813" 22:33:33 INFO [loop_until]: kubectl --namespace=xlou top node 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG --- status code --- 22:33:33 ERROR --- http response --- 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG --- http response --- 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713648813" 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG --- http response --- 22:33:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG --- status code --- 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:33:33 DEBUG { 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG ----- output ----- 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:33 DEBUG { 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648813" 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG { 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG --- http response --- 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG { 22:33:33 DEBUG { 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 DEBUG { 22:33:33 DEBUG { 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 DEBUG "status": "success", 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:33:33 DEBUG { 22:33:33 DEBUG "data": { 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG "status": "success", 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713648813" 22:33:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648813" 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG { 22:33:33 DEBUG "data": { 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG "data": { 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:33 DEBUG ----- output ----- 22:33:33 INFO [loop_until]: OK (rc = 0) 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG "data": { 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 DEBUG "data": { 22:33:33 DEBUG "data": { 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG "data": { 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG { 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG { 22:33:33 DEBUG --- stdout --- 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 DEBUG --- http response --- 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 DEBUG { 22:33:33 DEBUG "data": { 22:33:33 DEBUG "result": [ 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG { 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG "data": { 22:33:33 DEBUG "result": [ 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG "result": [ 22:33:33 DEBUG { 22:33:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 3329m 8714Mi am-767f6b67b5-qcqz2 5460m 8834Mi am-767f6b67b5-vbltz 4900m 8848Mi ds-cts-0 2597m 3093Mi ds-cts-1 1559m 12271Mi ds-cts-2 1403m 11767Mi ds-idrepo-0 1697m 3931Mi ds-idrepo-1 1373m 3997Mi ds-idrepo-2 2192m 3901Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 2m 61Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 5115m 3092Mi 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG { 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG "status": "success", 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 DEBUG "result": [ 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG "result": [ 22:33:33 DEBUG "result": [ 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG "result": [ 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG { 22:33:33 DEBUG "data": { 22:33:33 DEBUG { 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:33 DEBUG "status": "success", 22:33:33 INFO [http_cmd]: http status code OK 22:33:33 DEBUG "data": { 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG --- stderr --- 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG "data": { 22:33:33 DEBUG "result": [ 22:33:33 DEBUG "metric": { 22:33:33 DEBUG "data": { 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG { 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG { 22:33:33 DEBUG "metric": { 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:33 DEBUG { 22:33:33 DEBUG "result": [ 22:33:33 DEBUG "metric": { 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG "metric": { 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG "data": { 22:33:33 DEBUG --- status code --- 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG { 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG { 22:33:33 DEBUG "pod": "ds-idrepo-0" 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG "metric": { 22:33:33 DEBUG { 22:33:33 DEBUG "instance": "10.142.0.107:9100" 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG "metric": { 22:33:33 DEBUG { 22:33:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:33:33 DEBUG "result": [ 22:33:33 DEBUG "instance": "10.142.0.107:9100" 22:33:33 DEBUG { 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:33:33 DEBUG "result": [ 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:33:33 DEBUG "result": [ 22:33:33 DEBUG "metric": { 22:33:33 DEBUG }, 22:33:33 DEBUG "result": [ 22:33:33 DEBUG { 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG "pod": "ds-idrepo-0" 22:33:33 DEBUG "metric": { 22:33:33 DEBUG }, 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG { 22:33:33 DEBUG "pod": "am-767f6b67b5-9zw22" 22:33:33 DEBUG "metric": { 22:33:33 DEBUG }, 22:33:33 DEBUG { 22:33:33 DEBUG }, 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG "result": [ 22:33:33 DEBUG --- http response --- 22:33:33 DEBUG { 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG "data": { 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG { 22:33:33 DEBUG "pod": "ds-idrepo-0" 22:33:33 DEBUG "value": [ 22:33:33 DEBUG { 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:33:33 DEBUG }, 22:33:33 DEBUG "pod": "ds-cts-0" 22:33:33 DEBUG "value": [ 22:33:33 DEBUG { 22:33:33 DEBUG { 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG }, 22:33:33 DEBUG "pod": "am-767f6b67b5-9zw22" 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "metric": { 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "data": { 22:33:33 DEBUG { 22:33:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:33:33 DEBUG "metric": { 22:33:33 DEBUG { 22:33:33 DEBUG { 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG { 22:33:33 DEBUG "metric": { 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG "metric": { 22:33:33 DEBUG "data": { 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG "value": [ 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG "data": { 22:33:33 DEBUG "value": [ 22:33:33 DEBUG }, 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG "metric": { 22:33:33 DEBUG ----- output ----- 22:33:33 DEBUG "pod": "ds-cts-0" 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG "result": [ 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG "pod": "ds-idrepo-0" 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG "pod": "am-767f6b67b5-9zw22" 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG { 22:33:33 DEBUG 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG "data": { 22:33:33 DEBUG "data": { 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG 22:33:33 DEBUG "pod": "ds-idrepo-0" 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "result": [ 22:33:33 DEBUG "instance": "10.142.0.107:9100" 22:33:33 DEBUG { 22:33:33 DEBUG }, 22:33:33 DEBUG "data": { 22:33:33 DEBUG "data": { 22:33:33 DEBUG { 22:33:33 DEBUG "data": { 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG [...] 22:33:33 DEBUG "result": [ 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG "result": [ 22:33:33 DEBUG 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG [...] 22:33:33 DEBUG }, 22:33:33 DEBUG "status": "success", 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG "metric": { 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG { 22:33:33 DEBUG "data": { 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG [...] 22:33:33 DEBUG "result": [ 22:33:33 DEBUG "result": [ 22:33:33 DEBUG { 22:33:33 DEBUG 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG { 22:33:33 DEBUG [...] 22:33:33 DEBUG 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "data": { 22:33:33 DEBUG 22:33:33 DEBUG "result": [ 22:33:33 DEBUG "result": [ 22:33:33 DEBUG "pod": "am-767f6b67b5-9zw22" 22:33:33 DEBUG "result": [ 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG [...] 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG { 22:33:33 DEBUG { 22:33:33 DEBUG "metric": { 22:33:33 DEBUG [...] 22:33:33 DEBUG 22:33:33 DEBUG [...] 22:33:33 DEBUG "metric": { 22:33:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG "resultType": "vector", 22:33:33 DEBUG 22:33:33 DEBUG { 22:33:33 DEBUG { 22:33:33 DEBUG }, 22:33:33 DEBUG { 22:33:33 DEBUG 22:33:33 DEBUG [...] 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG }, 22:33:33 DEBUG "result": [ 22:33:33 DEBUG 22:33:33 DEBUG [...] 22:33:33 DEBUG 22:33:33 DEBUG "metric": { 22:33:33 DEBUG "metric": { 22:33:33 DEBUG "pod": "am-767f6b67b5-9zw22" 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG "result": [ 22:33:33 DEBUG 22:33:33 DEBUG "metric": { 22:33:33 DEBUG "metric": { 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "metric": { 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "metric": { 22:33:33 DEBUG "pod": "ds-idrepo-0" 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "instance": "10.142.0.107:9100" 22:33:33 DEBUG "pod": "am-767f6b67b5-9zw22" 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "pod": "am-767f6b67b5-vbltz" 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG { 22:33:33 DEBUG [...] 22:33:33 DEBUG "pod": "ds-cts-0" 22:33:33 DEBUG "pod": "ds-idrepo-0" 22:33:33 DEBUG 22:33:33 DEBUG "pod": "am-767f6b67b5-9zw22" 22:33:33 DEBUG [...] 22:33:33 DEBUG 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG { 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "instance": "10.142.0.43:9100" 22:33:33 DEBUG }, 22:33:33 DEBUG }, 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG [...] 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG [...] 22:33:33 DEBUG "metric": { 22:33:33 DEBUG 22:33:33 DEBUG }, 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "0" 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "metric": { 22:33:33 DEBUG [...] 22:33:33 DEBUG "pod": "ds-idrepo-2" 22:33:33 DEBUG 22:33:33 DEBUG }, 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "instance": "10.142.0.107:9100" 22:33:33 DEBUG 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG "pod": "ds-idrepo-0" 22:33:33 DEBUG 22:33:33 DEBUG }, 22:33:33 DEBUG "pod": "ds-cts-2" 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "pod": "am-767f6b67b5-vbltz" 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "instance": "10.142.0.43:9100" 22:33:33 DEBUG }, 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG [...] 22:33:33 DEBUG 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG "pod": "ds-cts-2" 22:33:33 DEBUG 22:33:33 DEBUG [...] 22:33:33 DEBUG 22:33:33 DEBUG "pod": "ds-idrepo-2" 22:33:33 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG }, 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "value": [ 22:33:33 DEBUG }, 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG }, 22:33:33 DEBUG [...] 22:33:33 DEBUG 22:33:33 DEBUG }, 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "0" 22:33:33 DEBUG ] 22:33:33 DEBUG 22:33:33 DEBUG "value": [ 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG }, 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG } 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "4131245401088" 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG "pod": "ds-idrepo-2" 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG 22:33:33 DEBUG } 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG "0" 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG ] 22:33:33 DEBUG [...] 22:33:33 DEBUG [...] 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG "0" 22:33:33 DEBUG }, 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG "1.350484305635505" 22:33:33 DEBUG "instance": "10.142.0.43:9100" 22:33:33 DEBUG ] 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG 22:33:33 DEBUG [...] 22:33:33 DEBUG [...] 22:33:33 DEBUG [...] 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG "0" 22:33:33 DEBUG ] 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG ] 22:33:33 DEBUG "0" 22:33:33 DEBUG } 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG ] 22:33:33 DEBUG 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "0.005333333333332273" 22:33:33 DEBUG ] 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 INFO [loop_until]: OK (rc = 0) 22:33:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:33:33 DEBUG 22:33:33 DEBUG "0.00004420888888888888" 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG [...] 22:33:33 DEBUG "pod": "am-767f6b67b5-vbltz" 22:33:33 DEBUG [...] 22:33:33 DEBUG } 22:33:33 DEBUG ] 22:33:33 DEBUG ] 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG } 22:33:33 DEBUG 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG "0" 22:33:33 DEBUG }, 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG --- stdout --- 22:33:33 DEBUG }, 22:33:33 DEBUG [...] 22:33:33 DEBUG ] 22:33:33 DEBUG "pod": "am-767f6b67b5-vbltz" 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG } 22:33:33 DEBUG 22:33:33 DEBUG "0" 22:33:33 DEBUG ] 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG "pod": "am-767f6b67b5-vbltz" 22:33:33 DEBUG ] 22:33:33 DEBUG "value": [ 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG } 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG 22:33:33 DEBUG ] 22:33:33 DEBUG "pod": "am-767f6b67b5-vbltz" 22:33:33 DEBUG 22:33:33 DEBUG } 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG } 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG "instance": "10.142.0.43:9100" 22:33:33 DEBUG "pod": "am-767f6b67b5-vbltz" 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG }, 22:33:33 DEBUG } 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6265Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2334m 7% 7701Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 5290m 17% 11449Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3401Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 2060m 6% 6318Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5202Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 3560m 11% 11729Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 3741m 12% 3180Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 5511m 18% 11087Mi 9% gke-xlou-cdm-ds-0f601751-j488 2837m 9% 7326Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 1918m 6% 17279Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 2599m 8% 7969Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 3047m 10% 14510Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 5560m 34% 11018Mi 18% 22:33:33 DEBUG } 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG 22:33:33 DEBUG ] 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "pod": "ds-idrepo-2" 22:33:33 DEBUG "pod": "ds-cts-2" 22:33:33 DEBUG } 22:33:33 DEBUG }, 22:33:33 DEBUG 22:33:33 DEBUG ] 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG }, 22:33:33 DEBUG }, 22:33:33 DEBUG ] 22:33:33 DEBUG "value": [ 22:33:33 DEBUG ] 22:33:33 DEBUG "6301911.566666666" 22:33:33 DEBUG --- stderr --- 22:33:33 DEBUG "3020967936" 22:33:33 DEBUG 22:33:33 DEBUG } 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG }, 22:33:33 DEBUG }, 22:33:33 DEBUG } 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG } 22:33:33 DEBUG "16509.5" 22:33:33 DEBUG "pod": "ds-cts-2" 22:33:33 DEBUG } 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "value": [ 22:33:33 DEBUG } 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG } 22:33:33 DEBUG ] 22:33:33 DEBUG 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG "101.875" 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "value": [ 22:33:33 DEBUG 22:33:33 DEBUG "1102.5" 22:33:33 DEBUG } 22:33:33 DEBUG ] 22:33:33 DEBUG }, 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG } 22:33:33 DEBUG "1102.7" 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG "instance": "10.142.0.43:9100" 22:33:33 DEBUG ] 22:33:33 DEBUG ] 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG "pod": "ds-idrepo-2" 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG "value": [ 22:33:33 DEBUG "6303979.466666667" 22:33:33 DEBUG "1101.4666666666667" 22:33:33 DEBUG ] 22:33:33 DEBUG ] 22:33:33 DEBUG }, 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG "0" 22:33:33 DEBUG "0" 22:33:33 DEBUG }, 22:33:33 DEBUG } 22:33:33 DEBUG ] 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG ] 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG "value": [ 22:33:33 DEBUG } 22:33:33 DEBUG ] 22:33:33 DEBUG ] 22:33:33 DEBUG ] 22:33:33 DEBUG "value": [ 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG "0" 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG 1713648813, 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG ] 22:33:33 DEBUG ] 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG "0.007999999999964301" 22:33:33 DEBUG } 22:33:33 DEBUG ] 22:33:33 DEBUG ] 22:33:33 DEBUG "0" 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG ] 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG ] 22:33:33 DEBUG ] 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG } 22:33:33 DEBUG } 22:34:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713648873" 22:34:33 ERROR [http_cmd]: ERROR 22:34:33 ERROR --- status code --- 22:34:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:34:33 ERROR --- http response --- 22:34:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:34:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:34:33 INFO 22:34:33 INFO [loop_until]: kubectl --namespace=xlou top pods 22:34:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:34:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648873" 22:34:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648873" 22:34:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648873" 22:34:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648873" 22:34:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648873" 22:34:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713648873" 22:34:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648873" 22:34:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648873" 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648873" 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG --- status code --- 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648874" 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG ----- output ----- 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG { 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG { 22:34:34 DEBUG { 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG { 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG { 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG "data": { 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG "data": { 22:34:34 DEBUG "data": { 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG "data": { 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG "data": { 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:34:34 DEBUG ----- output ----- 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG "result": [ 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG "result": [ 22:34:34 DEBUG "result": [ 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG "result": [ 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG { 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG { 22:34:34 DEBUG { 22:34:34 DEBUG { 22:34:34 DEBUG { 22:34:34 DEBUG "result": [ 22:34:34 DEBUG { 22:34:34 DEBUG { 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG "metric": { 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG "metric": { 22:34:34 DEBUG "metric": { 22:34:34 DEBUG { 22:34:34 DEBUG "metric": { 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG "data": { 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG "pod": "ds-idrepo-0" 22:34:34 DEBUG "data": { 22:34:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:34:34 DEBUG "pod": "ds-idrepo-0" 22:34:34 DEBUG "metric": { 22:34:34 DEBUG "pod": "ds-idrepo-0" 22:34:34 DEBUG "data": { 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:34:34 DEBUG }, 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG }, 22:34:34 DEBUG }, 22:34:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:34:34 DEBUG }, 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG "result": [ 22:34:34 DEBUG "value": [ 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG "result": [ 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "value": [ 22:34:34 DEBUG }, 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "result": [ 22:34:34 DEBUG { 22:34:34 DEBUG 22:34:34 DEBUG { 22:34:34 DEBUG { 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 22:34:34 DEBUG { 22:34:34 DEBUG "metric": { 22:34:34 DEBUG 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG "metric": { 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "metric": { 22:34:34 DEBUG "instance": "10.142.0.107:9100" 22:34:34 DEBUG 22:34:34 DEBUG "data": { 22:34:34 DEBUG "pod": "ds-idrepo-0" 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "pod": "ds-cts-0" 22:34:34 DEBUG }, 22:34:34 DEBUG [...] 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG }, 22:34:34 DEBUG [...] 22:34:34 DEBUG [...] 22:34:34 DEBUG 22:34:34 DEBUG [...] 22:34:34 DEBUG }, 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 22:34:34 DEBUG "result": [ 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG [...] 22:34:34 DEBUG 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG { 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "metric": { 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "pod": "ds-idrepo-2" 22:34:34 DEBUG "instance": "10.142.0.107:9100" 22:34:34 DEBUG 22:34:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:34:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:34:34 DEBUG 22:34:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:34:34 DEBUG 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648874" 22:34:34 DEBUG [...] 22:34:34 DEBUG }, 22:34:34 DEBUG }, 22:34:34 DEBUG [...] 22:34:34 DEBUG }, 22:34:34 DEBUG }, 22:34:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:34:34 DEBUG }, 22:34:34 DEBUG [...] 22:34:34 DEBUG 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "value": [ 22:34:34 DEBUG }, 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 1713648873, 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 1713648873, 22:34:34 DEBUG 1713648873, 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 1713648873, 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "instance": "10.142.0.43:9100" 22:34:34 DEBUG "0" 22:34:34 DEBUG 22:34:34 DEBUG "0" 22:34:34 DEBUG "0" 22:34:34 DEBUG 1713648873, 22:34:34 DEBUG "0" 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG }, 22:34:34 DEBUG ] 22:34:34 DEBUG "pod": "ds-idrepo-2" 22:34:34 DEBUG ] 22:34:34 DEBUG ] 22:34:34 DEBUG "101.875" 22:34:34 DEBUG ] 22:34:34 DEBUG "pod": "ds-cts-2" 22:34:34 DEBUG 22:34:34 DEBUG "value": [ 22:34:34 DEBUG } 22:34:34 DEBUG }, 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG }, 22:34:34 DEBUG [...] 22:34:34 DEBUG 1713648873, 22:34:34 DEBUG ] 22:34:34 DEBUG "value": [ 22:34:34 DEBUG ] 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 22:34:34 DEBUG "0.03699999999998909" 22:34:34 DEBUG } 22:34:34 DEBUG 1713648873, 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG 1713648873, 22:34:34 DEBUG 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG "0" 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG "0.00010508444444444445" 22:34:34 DEBUG 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG "instance": "10.142.0.43:9100" 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG }, 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG "value": [ 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG } 22:34:34 DEBUG "0.05439999999998027" 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG { 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG "data": { 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG "result": [ 22:34:34 DEBUG { 22:34:34 DEBUG "metric": { 22:34:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:34:34 DEBUG }, 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG [...] 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:34:34 DEBUG }, 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG "3333.7666666666664" 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648874" 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648874" 22:34:34 INFO [loop_until]: OK (rc = 0) 22:34:34 DEBUG --- stdout --- 22:34:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17454m 8728Mi am-767f6b67b5-qcqz2 18332m 8842Mi am-767f6b67b5-vbltz 18824m 8866Mi ds-cts-0 11181m 5379Mi ds-cts-1 8900m 9377Mi ds-cts-2 10362m 11798Mi ds-idrepo-0 8219m 3907Mi ds-idrepo-1 8095m 3997Mi ds-idrepo-2 8330m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 2m 61Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11600m 3147Mi 22:34:34 DEBUG --- stderr --- 22:34:34 DEBUG 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713648874" 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648874" 22:34:34 INFO 22:34:34 INFO [loop_until]: kubectl --namespace=xlou top node 22:34:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648874" 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713648874" 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648874" 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648874" 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG --- status code --- 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG { 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG { 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG { 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG { 22:34:34 DEBUG "status": "success", 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG "data": { 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG "data": { 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648874" 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG "data": { 22:34:34 DEBUG "data": { 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:34 DEBUG "resultType": "vector", 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648874" 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648874" 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648874" 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648874" 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648874" 22:34:34 DEBUG "result": [ 22:34:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713648874" 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG "result": [ 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG --- status code --- 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG { 22:34:34 DEBUG "result": [ 22:34:34 DEBUG "result": [ 22:34:34 DEBUG { 22:34:34 DEBUG { 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG "metric": { 22:34:34 DEBUG { 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG { 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG "metric": { 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:34:34 DEBUG "metric": { 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG "metric": { 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG "data": { 22:34:34 DEBUG "instance": "10.142.0.107:9100" 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG }, 22:34:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG "pod": "ds-cts-0" 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG }, 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG { 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:34 DEBUG "value": [ 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG }, 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG }, 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG "result": [ 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG { 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "value": [ 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG { 22:34:34 DEBUG { 22:34:34 DEBUG 22:34:34 DEBUG "data": { 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG { 22:34:34 DEBUG 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG "metric": { 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:34:34 INFO [http_cmd]: http status code OK 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG "data": { 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG { 22:34:34 DEBUG { 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:34:34 DEBUG "pod": "ds-idrepo-0" 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG --- status code --- 22:34:34 DEBUG "data": { 22:34:34 DEBUG [...] 22:34:34 DEBUG "result": [ 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG "data": { 22:34:34 DEBUG [...] 22:34:34 DEBUG 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:34:34 DEBUG 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG }, 22:34:34 DEBUG { 22:34:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG 22:34:34 DEBUG { 22:34:34 DEBUG "result": [ 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG 22:34:34 DEBUG [...] 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG [...] 22:34:34 DEBUG "data": { 22:34:34 DEBUG "data": { 22:34:34 DEBUG { 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG --- http response --- 22:34:34 DEBUG "result": [ 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "instance": "10.142.0.43:9100" 22:34:34 DEBUG { 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG { 22:34:34 DEBUG 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG 22:34:34 DEBUG "data": { 22:34:34 DEBUG { 22:34:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:34:34 DEBUG ----- output ----- 22:34:34 DEBUG "result": [ 22:34:34 DEBUG }, 22:34:34 DEBUG "metric": { 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG 22:34:34 DEBUG "result": [ 22:34:34 DEBUG "result": [ 22:34:34 DEBUG "data": { 22:34:34 DEBUG 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG "metric": { 22:34:34 DEBUG "instance": "10.142.0.107:9100" 22:34:34 DEBUG { 22:34:34 DEBUG { 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:34:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:34:34 DEBUG 22:34:34 DEBUG "data": { 22:34:34 DEBUG 22:34:34 DEBUG { 22:34:34 DEBUG { 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG 22:34:34 DEBUG "result": [ 22:34:34 DEBUG "metric": { 22:34:34 DEBUG }, 22:34:34 DEBUG "status": "success", 22:34:34 DEBUG "metric": { 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG }, 22:34:34 DEBUG }, 22:34:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG "pod": "ds-cts-2" 22:34:34 DEBUG "metric": { 22:34:34 DEBUG "metric": { 22:34:34 DEBUG "result": [ 22:34:34 DEBUG [...] 22:34:34 DEBUG { 22:34:34 DEBUG "metric": { 22:34:34 DEBUG "pod": "ds-idrepo-0" 22:34:34 DEBUG "data": { 22:34:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:34:34 DEBUG "4133611226112" 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "value": [ 22:34:34 DEBUG }, 22:34:34 DEBUG "result": [ 22:34:34 DEBUG }, 22:34:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:34:34 DEBUG "instance": "10.142.0.107:9100" 22:34:34 DEBUG { 22:34:34 DEBUG 22:34:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:34:34 DEBUG }, 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "resultType": "vector", 22:34:34 DEBUG }, 22:34:34 DEBUG ] 22:34:34 DEBUG 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG "value": [ 22:34:34 DEBUG { 22:34:34 DEBUG "value": [ 22:34:34 DEBUG }, 22:34:34 DEBUG }, 22:34:34 DEBUG "metric": { 22:34:34 DEBUG 22:34:34 DEBUG }, 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 22:34:34 DEBUG "result": [ 22:34:34 DEBUG "value": [ 22:34:34 DEBUG } 22:34:34 DEBUG 22:34:34 DEBUG "0" 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG "metric": { 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG "0" 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "pod": "ds-idrepo-0" 22:34:34 DEBUG 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG { 22:34:34 DEBUG 22:34:34 DEBUG ] 22:34:34 DEBUG 22:34:34 DEBUG ] 22:34:34 DEBUG "11.735092153921288" 22:34:34 DEBUG "pod": "ds-cts-0" 22:34:34 DEBUG }, 22:34:34 DEBUG ] 22:34:34 DEBUG 22:34:34 DEBUG }, 22:34:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "metric": { 22:34:34 DEBUG 22:34:34 DEBUG } 22:34:34 DEBUG [...] 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "value": [ 22:34:34 DEBUG } 22:34:34 DEBUG 22:34:34 DEBUG "value": [ 22:34:34 DEBUG }, 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG [...] 22:34:34 DEBUG "pod": "ds-idrepo-0" 22:34:34 DEBUG 22:34:34 DEBUG } 22:34:34 DEBUG 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG 22:34:34 DEBUG ] 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 22:34:34 DEBUG [...] 22:34:34 DEBUG 22:34:34 DEBUG }, 22:34:34 DEBUG [...] 22:34:34 DEBUG 22:34:34 DEBUG } 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG } 22:34:34 DEBUG [...] 22:34:34 DEBUG 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG [...] 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG } 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG } 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "0" 22:34:34 DEBUG ] 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG } 22:34:34 DEBUG 22:34:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:34:34 DEBUG } 22:34:34 DEBUG [...] 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG [...] 22:34:34 DEBUG 22:34:34 DEBUG } 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:34:34 DEBUG }, 22:34:34 DEBUG [...] 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG ] 22:34:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:34:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:34:34 DEBUG 22:34:34 DEBUG }, 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 22:34:34 DEBUG 22:34:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:34:34 DEBUG 22:34:34 DEBUG "instance": "10.142.0.43:9100" 22:34:34 DEBUG } 22:34:34 DEBUG }, 22:34:34 DEBUG }, 22:34:34 DEBUG [...] 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG 22:34:34 DEBUG "instance": "10.142.0.43:9100" 22:34:34 DEBUG }, 22:34:34 DEBUG 22:34:34 DEBUG }, 22:34:34 DEBUG } 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG "3394265088" 22:34:34 DEBUG 22:34:34 DEBUG }, 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "pod": "ds-idrepo-2" 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "0.00001756690379009013" 22:34:34 DEBUG 22:34:34 DEBUG "3333.0333333333333" 22:34:34 DEBUG ] 22:34:34 DEBUG "pod": "ds-cts-2" 22:34:34 DEBUG "value": [ 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG }, 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG ] 22:34:34 DEBUG 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG }, 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG "3333.1" 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "79561585.19999999" 22:34:34 DEBUG "181393.5" 22:34:34 DEBUG } 22:34:34 DEBUG "pod": "ds-cts-2" 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG "value": [ 22:34:34 DEBUG "79561036.3" 22:34:34 DEBUG ] 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG ] 22:34:34 DEBUG ] 22:34:34 DEBUG ] 22:34:34 DEBUG }, 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG "0" 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG "value": [ 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG "0" 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG 1713648874, 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG "0" 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG ] 22:34:34 DEBUG ] 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 DEBUG } 22:34:34 INFO [loop_until]: OK (rc = 0) 22:34:34 DEBUG --- stdout --- 22:34:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6284Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8674m 29% 7663Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17465m 58% 11445Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3390Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 11381m 38% 8344Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5196Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 18118m 60% 11728Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19392m 64% 3162Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 20142m 67% 11085Mi 9% gke-xlou-cdm-ds-0f601751-j488 9165m 30% 7276Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 8824m 29% 14435Mi 12% gke-xlou-cdm-ds-0f601751-nj6q 9075m 30% 7975Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 10119m 33% 14559Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 12017m 75% 11071Mi 18% 22:34:34 DEBUG --- stderr --- 22:34:34 DEBUG 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713648934" 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 ERROR [http_cmd]: ERROR 22:35:34 ERROR --- status code --- 22:35:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:35:34 ERROR --- http response --- 22:35:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:35:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:35:34 INFO 22:35:34 INFO [loop_until]: kubectl --namespace=xlou top pods 22:35:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG { 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG "data": { 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG "result": [ 22:35:34 DEBUG { 22:35:34 DEBUG "metric": { 22:35:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:35:34 DEBUG }, 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG [...] 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:35:34 DEBUG }, 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG "0" 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG { 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG "data": { 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG "result": [ 22:35:34 DEBUG { 22:35:34 DEBUG "metric": { 22:35:34 DEBUG "pod": "ds-idrepo-0" 22:35:34 DEBUG }, 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG [...] 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "pod": "ds-idrepo-2" 22:35:34 DEBUG }, 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG "0" 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648934" 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648934" 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG --- status code --- 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG --- status code --- 22:35:34 INFO [loop_until]: OK (rc = 0) 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG --- stdout --- 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG --- http response --- 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17421m 8728Mi am-767f6b67b5-qcqz2 18177m 8844Mi am-767f6b67b5-vbltz 18985m 8868Mi ds-cts-0 10610m 8233Mi ds-cts-1 9511m 9417Mi ds-cts-2 10278m 11822Mi ds-idrepo-0 8267m 3907Mi ds-idrepo-1 8412m 3997Mi ds-idrepo-2 8254m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11542m 3154Mi 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG --- stderr --- 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG { 22:35:34 DEBUG 22:35:34 DEBUG { 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG { 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:34 DEBUG "data": { 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG "data": { 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG "data": { 22:35:34 DEBUG { 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:35:34 DEBUG { 22:35:34 DEBUG "result": [ 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG "result": [ 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG { 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG { 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG "result": [ 22:35:34 DEBUG "data": { 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG { 22:35:34 DEBUG "data": { 22:35:34 DEBUG "metric": { 22:35:34 DEBUG "metric": { 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 DEBUG { 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG "pod": "ds-idrepo-0" 22:35:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG "metric": { 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG "result": [ 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG "data": { 22:35:34 DEBUG "result": [ 22:35:34 DEBUG }, 22:35:34 DEBUG }, 22:35:34 DEBUG { 22:35:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:35:34 DEBUG { 22:35:34 DEBUG { 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG { 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG }, 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG "metric": { 22:35:34 DEBUG "result": [ 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:34 DEBUG "metric": { 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "data": { 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "data": { 22:35:34 DEBUG "pod": "ds-cts-0" 22:35:34 DEBUG { 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG }, 22:35:34 DEBUG "metric": { 22:35:34 DEBUG { 22:35:34 DEBUG }, 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "result": [ 22:35:34 DEBUG 22:35:34 DEBUG "result": [ 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "pod": "ds-cts-0" 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG "value": [ 22:35:34 DEBUG [...] 22:35:34 DEBUG [...] 22:35:34 DEBUG { 22:35:34 DEBUG 22:35:34 DEBUG { 22:35:34 DEBUG 22:35:34 DEBUG }, 22:35:34 DEBUG "data": { 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "metric": { 22:35:34 DEBUG [...] 22:35:34 DEBUG "metric": { 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "instance": "10.142.0.107:9100" 22:35:34 DEBUG 22:35:34 DEBUG "instance": "10.142.0.107:9100" 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "result": [ 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG }, 22:35:34 DEBUG 22:35:34 DEBUG }, 22:35:34 DEBUG [...] 22:35:34 DEBUG 22:35:34 DEBUG { 22:35:34 DEBUG [...] 22:35:34 DEBUG "pod": "ds-idrepo-2" 22:35:34 DEBUG "pod": "idm-b85f9d978-wnhxb" 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "metric": { 22:35:34 DEBUG 22:35:34 DEBUG }, 22:35:34 DEBUG }, 22:35:34 DEBUG 22:35:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG [...] 22:35:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG }, 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG }, 22:35:34 DEBUG 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG "pod": "ds-cts-2" 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "pod": "idm-b85f9d978-wnhxb" 22:35:34 DEBUG "0" 22:35:34 DEBUG "47621473.63812164" 22:35:34 DEBUG [...] 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG [...] 22:35:34 DEBUG }, 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG }, 22:35:34 DEBUG ] 22:35:34 DEBUG ] 22:35:34 DEBUG 22:35:34 DEBUG "101.875" 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "pod": "ds-cts-2" 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG 22:35:34 DEBUG ] 22:35:34 DEBUG 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG }, 22:35:34 DEBUG 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG ] 22:35:34 DEBUG ] 22:35:34 DEBUG 22:35:34 DEBUG } 22:35:34 DEBUG 22:35:34 DEBUG "0" 22:35:34 DEBUG "value": [ 22:35:34 DEBUG [...] 22:35:34 DEBUG "0" 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG "instance": "10.142.0.43:9100" 22:35:34 DEBUG ] 22:35:34 DEBUG "instance": "10.142.0.43:9100" 22:35:34 DEBUG ] 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG }, 22:35:34 DEBUG } 22:35:34 DEBUG }, 22:35:34 DEBUG } 22:35:34 DEBUG "0" 22:35:34 DEBUG 22:35:34 DEBUG } 22:35:34 DEBUG "value": [ 22:35:34 DEBUG } 22:35:34 DEBUG "value": [ 22:35:34 DEBUG ] 22:35:34 DEBUG ] 22:35:34 DEBUG 22:35:34 DEBUG ] 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:35:34 DEBUG } 22:35:34 DEBUG "0.061300000000058696" 22:35:34 DEBUG "0.03999999999999962" 22:35:34 DEBUG } 22:35:34 DEBUG ] 22:35:34 DEBUG }, 22:35:34 DEBUG } 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG ] 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG "value": [ 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG ] 22:35:34 DEBUG ] 22:35:34 DEBUG "3340.4" 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG ] 22:35:34 DEBUG { 22:35:34 DEBUG } 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG } 22:35:34 DEBUG "data": { 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG "result": [ 22:35:34 DEBUG { 22:35:34 DEBUG "metric": { 22:35:34 DEBUG "pod": "ds-idrepo-0" 22:35:34 DEBUG }, 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG [...] 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "pod": "ds-cts-2" 22:35:34 DEBUG }, 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG "0" 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648934" 22:35:34 INFO 22:35:34 INFO [loop_until]: kubectl --namespace=xlou top node 22:35:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648934" 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG --- status code --- 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713648934" 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713648934" 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG --- http response --- 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG ----- output ----- 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 DEBUG { 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG { 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648934" 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 DEBUG --- status code --- 22:35:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648934" 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG "status": "success", 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG "data": { 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG "data": { 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG "resultType": "vector", 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG "result": [ 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:34 DEBUG "result": [ 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:34 DEBUG { 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG { 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG { 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG ----- output ----- 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG "metric": { 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG { 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG "metric": { 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG { 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG { 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG { 22:35:34 DEBUG "data": { 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:34 DEBUG "data": { 22:35:34 INFO [http_cmd]: http status code OK 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG { 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG }, 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:34 DEBUG "instance": "10.142.0.107:9100" 22:35:34 DEBUG "result": [ 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG --- status code --- 22:35:34 DEBUG "data": { 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG "data": { 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "data": { 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG }, 22:35:34 DEBUG { 22:35:34 DEBUG { 22:35:34 DEBUG "result": [ 22:35:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG "data": { 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:35:34 DEBUG 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG { 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "metric": { 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG { 22:35:34 DEBUG --- http response --- 22:35:34 DEBUG "result": [ 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG "result": [ 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG 22:35:34 DEBUG "result": [ 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG 22:35:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:35:34 DEBUG "data": { 22:35:34 DEBUG "metric": { 22:35:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:35:34 DEBUG { 22:35:34 DEBUG "result": [ 22:35:34 DEBUG { 22:35:34 DEBUG { 22:35:34 DEBUG { 22:35:34 DEBUG 22:35:34 DEBUG { 22:35:34 DEBUG "data": { 22:35:34 DEBUG 22:35:34 DEBUG }, 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:35:34 DEBUG ----- output ----- 22:35:34 DEBUG "metric": { 22:35:34 DEBUG { 22:35:34 DEBUG "metric": { 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG [...] 22:35:34 DEBUG "metric": { 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "result": [ 22:35:34 DEBUG }, 22:35:34 DEBUG { 22:35:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:35:34 DEBUG "metric": { 22:35:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:35:34 DEBUG "data": { 22:35:34 DEBUG "data": { 22:35:34 DEBUG 22:35:34 DEBUG "instance": "10.142.0.107:9100" 22:35:34 DEBUG "result": [ 22:35:34 DEBUG [...] 22:35:34 DEBUG 22:35:34 DEBUG { 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "status": "success", 22:35:34 DEBUG }, 22:35:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:35:34 DEBUG }, 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG 22:35:34 DEBUG }, 22:35:34 DEBUG { 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "metric": { 22:35:34 DEBUG 22:35:34 DEBUG "data": { 22:35:34 DEBUG "value": [ 22:35:34 DEBUG }, 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "result": [ 22:35:34 DEBUG "result": [ 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "metric": { 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:35:34 DEBUG 22:35:34 DEBUG "resultType": "vector", 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG { 22:35:34 DEBUG { 22:35:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:35:34 DEBUG 22:35:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:35:34 DEBUG 22:35:34 DEBUG [...] 22:35:34 DEBUG }, 22:35:34 DEBUG 22:35:34 DEBUG "result": [ 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "metric": { 22:35:34 DEBUG "metric": { 22:35:34 DEBUG }, 22:35:34 DEBUG 22:35:34 DEBUG }, 22:35:34 DEBUG "instance": "10.142.0.43:9100" 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG [...] 22:35:34 DEBUG { 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "pod": "ds-idrepo-0" 22:35:34 DEBUG "instance": "10.142.0.107:9100" 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG }, 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "metric": { 22:35:34 DEBUG [...] 22:35:34 DEBUG 22:35:34 DEBUG [...] 22:35:34 DEBUG }, 22:35:34 DEBUG }, 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG [...] 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "pod": "ds-cts-0" 22:35:34 DEBUG 22:35:34 DEBUG [...] 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "value": [ 22:35:34 DEBUG "3339.9666666666667" 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG }, 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG ] 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "79900247.53333333" 22:35:34 DEBUG }, 22:35:34 DEBUG [...] 22:35:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG } 22:35:34 DEBUG 22:35:34 DEBUG [...] 22:35:34 DEBUG ] 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG }, 22:35:34 DEBUG 22:35:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:35:34 DEBUG 22:35:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG ] 22:35:34 DEBUG "instance": "10.142.0.43:9100" 22:35:34 DEBUG 22:35:34 DEBUG } 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG }, 22:35:34 DEBUG "pod": "idm-b85f9d978-wnhxb" 22:35:34 DEBUG }, 22:35:34 DEBUG [...] 22:35:34 DEBUG [...] 22:35:34 DEBUG } 22:35:34 DEBUG }, 22:35:34 DEBUG 22:35:34 DEBUG ] 22:35:34 DEBUG "11.6568777537499" 22:35:34 DEBUG 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG 22:35:34 DEBUG "value": [ 22:35:34 DEBUG }, 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG } 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 22:35:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:35:34 DEBUG }, 22:35:34 DEBUG "pod": "idm-b85f9d978-wnhxb" 22:35:34 DEBUG "3340.133333333333" 22:35:34 DEBUG [...] 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG 22:35:34 DEBUG 22:35:34 DEBUG "pod": "ds-idrepo-2" 22:35:34 DEBUG }, 22:35:34 DEBUG } 22:35:34 DEBUG "value": [ 22:35:34 DEBUG }, 22:35:34 DEBUG ] 22:35:34 DEBUG 22:35:34 DEBUG "0" 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG "3402055680" 22:35:34 DEBUG 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG ] 22:35:34 DEBUG "value": [ 22:35:34 DEBUG } 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG "value": [ 22:35:34 DEBUG } 22:35:34 DEBUG 22:35:34 DEBUG ] 22:35:34 DEBUG "448.35261064691497" 22:35:34 DEBUG ] 22:35:34 DEBUG 22:35:34 DEBUG "4137218219008" 22:35:34 DEBUG } 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG "0" 22:35:34 DEBUG "381584.5" 22:35:34 DEBUG ] 22:35:34 DEBUG 22:35:34 DEBUG } 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG "instance": "10.142.0.43:9100" 22:35:34 DEBUG ] 22:35:34 DEBUG ] 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG ] 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG "pod": "ds-cts-2" 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG ] 22:35:34 DEBUG }, 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG "0.0028822563578635397" 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG ] 22:35:34 DEBUG }, 22:35:34 DEBUG } 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG "value": [ 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG ] 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG "value": [ 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG "0" 22:35:34 DEBUG ] 22:35:34 DEBUG 1713648934, 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG "79900552.33333334" 22:35:34 DEBUG } 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG ] 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 DEBUG ] 22:35:34 DEBUG } 22:35:34 DEBUG } 22:35:34 INFO [loop_until]: OK (rc = 0) 22:35:34 DEBUG --- stdout --- 22:35:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6289Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8698m 29% 7676Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 17022m 57% 11460Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3391Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 10800m 36% 11143Mi 9% gke-xlou-cdm-default-pool-1b0769c8-wpwl 169m 0% 5206Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 18145m 60% 11727Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19511m 65% 3145Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 20214m 67% 11099Mi 9% gke-xlou-cdm-ds-0f601751-j488 9228m 30% 7293Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 9626m 32% 14456Mi 12% gke-xlou-cdm-ds-0f601751-nj6q 9115m 30% 7972Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 10243m 34% 14567Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11948m 75% 11075Mi 18% 22:35:34 DEBUG --- stderr --- 22:35:34 DEBUG 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 ERROR [http_cmd]: ERROR 22:36:34 ERROR --- status code --- 22:36:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:36:34 ERROR --- http response --- 22:36:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:36:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG { 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "data": { 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "result": [ 22:36:34 DEBUG { 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "pod": "ds-idrepo-0" 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG [...] 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "pod": "ds-idrepo-2" 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG "0" 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG { 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "data": { 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "result": [ 22:36:34 DEBUG { 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG [...] 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG "0" 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 INFO 22:36:34 INFO [loop_until]: kubectl --namespace=xlou top pods 22:36:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG { 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "data": { 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "result": [ 22:36:34 DEBUG { 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "pod": "ds-cts-0" 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG [...] 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "pod": "ds-cts-2" 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG "0.0006058266666666666" 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648994" 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG --- http response --- 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG --- http response --- 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG --- status code --- 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG { 22:36:34 DEBUG { 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG "data": { 22:36:34 DEBUG "data": { 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG { 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG "result": [ 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG "result": [ 22:36:34 DEBUG { 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG { 22:36:34 DEBUG { 22:36:34 DEBUG { 22:36:34 DEBUG { 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "data": { 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "data": { 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "data": { 22:36:34 DEBUG "pod": "ds-idrepo-0" 22:36:34 DEBUG "data": { 22:36:34 DEBUG "instance": "10.142.0.107:9100" 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG "result": [ 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG }, 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG }, 22:36:34 DEBUG "result": [ 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG { 22:36:34 DEBUG "result": [ 22:36:34 DEBUG "value": [ 22:36:34 DEBUG "result": [ 22:36:34 DEBUG "value": [ 22:36:34 DEBUG { 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG "metric": { 22:36:34 DEBUG { 22:36:34 DEBUG 22:36:34 DEBUG { 22:36:34 DEBUG 22:36:34 DEBUG "metric": { 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:36:34 DEBUG "pod": "ds-idrepo-0" 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:34 DEBUG "metric": { 22:36:34 DEBUG 22:36:34 DEBUG "metric": { 22:36:34 DEBUG 22:36:34 DEBUG "pod": "ds-idrepo-0" 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG }, 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:36:34 DEBUG 22:36:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:36:34 DEBUG 22:36:34 DEBUG }, 22:36:34 DEBUG { 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:36:34 DEBUG "value": [ 22:36:34 INFO [loop_until]: OK (rc = 0) 22:36:34 DEBUG { 22:36:34 DEBUG }, 22:36:34 DEBUG [...] 22:36:34 DEBUG }, 22:36:34 DEBUG [...] 22:36:34 DEBUG "value": [ 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG 22:36:34 DEBUG --- stdout --- 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "data": { 22:36:34 DEBUG { 22:36:34 DEBUG 22:36:34 DEBUG "data": { 22:36:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 19410m 8751Mi am-767f6b67b5-qcqz2 18478m 8849Mi am-767f6b67b5-vbltz 20407m 8892Mi ds-cts-0 15209m 11039Mi ds-cts-1 11534m 10406Mi ds-cts-2 12974m 11829Mi ds-idrepo-0 8760m 3907Mi ds-idrepo-1 8732m 3997Mi ds-idrepo-2 9026m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11766m 3161Mi 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG --- stderr --- 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "result": [ 22:36:34 DEBUG "data": { 22:36:34 DEBUG [...] 22:36:34 DEBUG "result": [ 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "pod": "ds-idrepo-2" 22:36:34 DEBUG 22:36:34 DEBUG "instance": "10.142.0.43:9100" 22:36:34 DEBUG [...] 22:36:34 DEBUG { 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG 22:36:34 DEBUG { 22:36:34 DEBUG [...] 22:36:34 DEBUG }, 22:36:34 DEBUG [...] 22:36:34 DEBUG }, 22:36:34 DEBUG 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "result": [ 22:36:34 DEBUG 22:36:34 DEBUG "metric": { 22:36:34 DEBUG 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG "pod": "ds-cts-0" 22:36:34 DEBUG { 22:36:34 DEBUG 22:36:34 DEBUG "instance": "10.142.0.107:9100" 22:36:34 DEBUG 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG 22:36:34 DEBUG }, 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "pod": "overseer-0-967d957f7-zk66v" 22:36:34 DEBUG }, 22:36:34 DEBUG 22:36:34 DEBUG "0" 22:36:34 DEBUG 22:36:34 DEBUG "0.042333333333336526" 22:36:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:36:34 DEBUG "value": [ 22:36:34 DEBUG "pod": "ds-idrepo-0" 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:36:34 DEBUG ] 22:36:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:36:34 DEBUG ] 22:36:34 DEBUG }, 22:36:34 DEBUG 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG }, 22:36:34 DEBUG } 22:36:34 DEBUG }, 22:36:34 DEBUG } 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG 22:36:34 DEBUG "value": [ 22:36:34 DEBUG ] 22:36:34 DEBUG "value": [ 22:36:34 DEBUG ] 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "0" 22:36:34 DEBUG 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG } 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG } 22:36:34 DEBUG "0" 22:36:34 DEBUG [...] 22:36:34 DEBUG 22:36:34 DEBUG ] 22:36:34 DEBUG [...] 22:36:34 DEBUG "3346.1" 22:36:34 DEBUG } 22:36:34 DEBUG "101.875" 22:36:34 DEBUG } 22:36:34 DEBUG ] 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG } 22:36:34 DEBUG 22:36:34 DEBUG ] 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG 22:36:34 DEBUG [...] 22:36:34 DEBUG ] 22:36:34 DEBUG 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG ] 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713648994" 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG } 22:36:34 DEBUG 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG "pod": "ds-cts-2" 22:36:34 DEBUG 22:36:34 DEBUG } 22:36:34 DEBUG "instance": "10.142.0.43:9100" 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG }, 22:36:34 DEBUG 22:36:34 DEBUG }, 22:36:34 DEBUG } 22:36:34 DEBUG "value": [ 22:36:34 DEBUG "pod": "ds-cts-2" 22:36:34 DEBUG "value": [ 22:36:34 DEBUG } 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG }, 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG "0" 22:36:34 DEBUG "value": [ 22:36:34 DEBUG "0.061866666666752695" 22:36:34 DEBUG ] 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG "0" 22:36:34 DEBUG } 22:36:34 DEBUG ] 22:36:34 DEBUG ] 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG --- http response --- 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG { 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "data": { 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "result": [ 22:36:34 DEBUG { 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG [...] 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG "1092.8666666666666" 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG { 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "data": { 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "result": [ 22:36:34 DEBUG { 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG [...] 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG "3408826368" 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG { 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "data": { 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "result": [ 22:36:34 DEBUG { 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "pod": "ds-idrepo-0" 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG [...] 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG "0" 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG { 22:36:34 DEBUG { 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "data": { 22:36:34 DEBUG "data": { 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "result": [ 22:36:34 DEBUG "result": [ 22:36:34 DEBUG { 22:36:34 DEBUG { 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "instance": "10.142.0.107:9100" 22:36:34 DEBUG "instance": "10.142.0.107:9100" 22:36:34 DEBUG }, 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:34 DEBUG 22:36:34 DEBUG [...] 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG [...] 22:36:34 DEBUG 22:36:34 DEBUG { 22:36:34 DEBUG { 22:36:34 DEBUG { 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "data": { 22:36:34 DEBUG "data": { 22:36:34 DEBUG "data": { 22:36:34 DEBUG 22:36:34 DEBUG "instance": "10.142.0.43:9100" 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "instance": "10.142.0.43:9100" 22:36:34 DEBUG }, 22:36:34 DEBUG "result": [ 22:36:34 DEBUG "result": [ 22:36:34 DEBUG "result": [ 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG { 22:36:34 DEBUG { 22:36:34 DEBUG { 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "metric": { 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG "81565959.4" 22:36:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:36:34 DEBUG "pod": "ds-cts-0" 22:36:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:36:34 DEBUG "81567476.26666667" 22:36:34 DEBUG ] 22:36:34 DEBUG }, 22:36:34 DEBUG }, 22:36:34 DEBUG }, 22:36:34 DEBUG ] 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713648994" 22:36:34 DEBUG } 22:36:34 DEBUG "value": [ 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG "value": [ 22:36:34 DEBUG ] 22:36:34 DEBUG 22:36:34 INFO 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 DEBUG } 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG } 22:36:34 DEBUG 22:36:34 INFO [loop_until]: kubectl --namespace=xlou top node 22:36:34 DEBUG ] 22:36:34 DEBUG 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713648994" 22:36:34 DEBUG 22:36:34 DEBUG } 22:36:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713648994" 22:36:34 DEBUG 22:36:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:36:34 DEBUG } 22:36:34 DEBUG [...] 22:36:34 DEBUG 22:36:34 DEBUG [...] 22:36:34 DEBUG } 22:36:34 DEBUG 22:36:34 DEBUG [...] 22:36:34 DEBUG 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG 22:36:34 DEBUG --- status code --- 22:36:34 INFO [http_cmd]: http status code OK 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG "pod": "ds-cts-2" 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG --- status code --- 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG }, 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG }, 22:36:34 DEBUG --- http response --- 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:34 DEBUG "value": [ 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG "value": [ 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:36:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG "0" 22:36:34 DEBUG { 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG ----- output ----- 22:36:34 DEBUG { 22:36:34 DEBUG "3350.0666666666666" 22:36:34 DEBUG { 22:36:34 DEBUG ] 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "3346.0333333333333" 22:36:34 DEBUG { 22:36:34 DEBUG { 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG ] 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG } 22:36:34 DEBUG "data": { 22:36:34 DEBUG ] 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "status": "success", 22:36:34 DEBUG "data": { 22:36:34 DEBUG } 22:36:34 DEBUG "data": { 22:36:34 DEBUG ] 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG } 22:36:34 DEBUG "data": { 22:36:34 DEBUG "data": { 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG ] 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG } 22:36:34 DEBUG "result": [ 22:36:34 DEBUG ] 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "resultType": "vector", 22:36:34 DEBUG "result": [ 22:36:34 DEBUG } 22:36:34 DEBUG "result": [ 22:36:34 DEBUG } 22:36:34 DEBUG { 22:36:34 DEBUG } 22:36:34 DEBUG "result": [ 22:36:34 DEBUG "result": [ 22:36:34 DEBUG { 22:36:34 DEBUG } 22:36:34 DEBUG { 22:36:34 DEBUG "metric": { 22:36:34 DEBUG } 22:36:34 DEBUG { 22:36:34 DEBUG { 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "metric": { 22:36:34 DEBUG "pod": "ds-idrepo-0" 22:36:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:36:34 DEBUG }, 22:36:34 DEBUG "pod": "ds-idrepo-0" 22:36:34 DEBUG "instance": "10.142.0.107:9100" 22:36:34 DEBUG }, 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG }, 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG "value": [ 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG [...] 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG [...] 22:36:34 DEBUG [...] 22:36:34 DEBUG 22:36:34 DEBUG [...] 22:36:34 DEBUG [...] 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:36:34 DEBUG 22:36:34 DEBUG 22:36:34 DEBUG "pod": "ds-idrepo-2" 22:36:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:36:34 DEBUG }, 22:36:34 DEBUG "pod": "overseer-0-967d957f7-zk66v" 22:36:34 DEBUG "instance": "10.142.0.43:9100" 22:36:34 DEBUG }, 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG }, 22:36:34 DEBUG }, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG "value": [ 22:36:34 DEBUG "value": [ 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG "11.673083416372808" 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG 1713648994, 22:36:34 DEBUG "0" 22:36:34 DEBUG "565677.5" 22:36:34 DEBUG ] 22:36:34 DEBUG "0" 22:36:34 DEBUG "4140875215872" 22:36:34 DEBUG ] 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG ] 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG ] 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG ] 22:36:34 DEBUG ] 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 DEBUG } 22:36:34 INFO [loop_until]: OK (rc = 0) 22:36:34 DEBUG --- stdout --- 22:36:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6288Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9122m 30% 7669Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 19206m 64% 11481Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3394Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 14292m 47% 13913Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 5207Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 19166m 64% 11742Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19691m 65% 3148Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 21698m 72% 11117Mi 9% gke-xlou-cdm-ds-0f601751-j488 9664m 32% 7274Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 13055m 43% 15613Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 9144m 30% 7983Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 15124m 50% 14511Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 11953m 75% 11130Mi 18% 22:36:34 DEBUG --- stderr --- 22:36:34 DEBUG 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 ERROR [http_cmd]: ERROR 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 ERROR --- status code --- 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 ERROR --- http response --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:37:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:37:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "pod": "ds-cts-0" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "pod": "ds-cts-2" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "0.3896666666666667" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "pod": "ds-idrepo-0" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "pod": "ds-idrepo-2" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "0" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "0" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649054" 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "instance": "10.142.0.107:9100" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "instance": "10.142.0.43:9100" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "0.0519999999999925" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG [...] 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:34 DEBUG }, 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG "value": [ 22:37:34 DEBUG { 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "3328.366666666667" 22:37:34 DEBUG "data": { 22:37:34 DEBUG ] 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG } 22:37:34 DEBUG "result": [ 22:37:34 DEBUG ] 22:37:34 DEBUG { 22:37:34 DEBUG } 22:37:34 DEBUG "metric": { 22:37:34 DEBUG } 22:37:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "3600977920" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649054" 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "pod": "ds-cts-1" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "36756075.84889019" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "0" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "instance": "10.142.0.107:9100" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "instance": "10.142.0.43:9100" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "0.1773999999999224" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO 22:37:34 INFO [loop_until]: kubectl --namespace=xlou top pods 22:37:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "pod": "ds-idrepo-0" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "pod": "ds-idrepo-2" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "0" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG { 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG "status": "success", 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG "data": { 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG "data": { 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG "result": [ 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:37:34 DEBUG { 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:37:34 DEBUG "metric": { 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG "metric": { 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:37:34 DEBUG { 22:37:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:37:34 DEBUG { 22:37:34 DEBUG }, 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG }, 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "value": [ 22:37:34 DEBUG "data": { 22:37:34 DEBUG "value": [ 22:37:34 DEBUG "data": { 22:37:34 DEBUG 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG 22:37:34 DEBUG "result": [ 22:37:34 DEBUG 22:37:34 DEBUG "result": [ 22:37:34 DEBUG 22:37:34 DEBUG { 22:37:34 DEBUG 22:37:34 DEBUG { 22:37:34 DEBUG [...] 22:37:34 DEBUG "metric": { 22:37:34 DEBUG [...] 22:37:34 DEBUG "metric": { 22:37:34 DEBUG 22:37:34 DEBUG "pod": "ds-cts-0" 22:37:34 DEBUG 22:37:34 DEBUG "pod": "ds-idrepo-0" 22:37:34 DEBUG 22:37:34 DEBUG }, 22:37:34 DEBUG 22:37:34 DEBUG }, 22:37:34 DEBUG 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG "value": [ 22:37:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:37:34 DEBUG 22:37:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:37:34 DEBUG 22:37:34 DEBUG }, 22:37:34 DEBUG 22:37:34 DEBUG }, 22:37:34 DEBUG 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG [...] 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG [...] 22:37:34 DEBUG "101.875" 22:37:34 DEBUG 22:37:34 DEBUG "3332.0333333333333" 22:37:34 DEBUG 22:37:34 DEBUG ] 22:37:34 DEBUG 22:37:34 DEBUG ] 22:37:34 DEBUG 22:37:34 DEBUG } 22:37:34 DEBUG 22:37:34 DEBUG } 22:37:34 DEBUG 22:37:34 DEBUG ] 22:37:34 DEBUG "pod": "ds-cts-2" 22:37:34 DEBUG ] 22:37:34 DEBUG "pod": "ds-cts-2" 22:37:34 DEBUG } 22:37:34 DEBUG }, 22:37:34 DEBUG } 22:37:34 DEBUG }, 22:37:34 DEBUG } 22:37:34 DEBUG "value": [ 22:37:34 DEBUG } 22:37:34 DEBUG "value": [ 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "0" 22:37:34 DEBUG "0" 22:37:34 DEBUG ] 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG { 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "pod": "ds-cts-0" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:34 DEBUG 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG 22:37:34 DEBUG { 22:37:34 DEBUG 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "pod": "ds-cts-2" 22:37:34 DEBUG "data": { 22:37:34 DEBUG }, 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "value": [ 22:37:34 DEBUG "result": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG { 22:37:34 DEBUG "0" 22:37:34 DEBUG "metric": { 22:37:34 DEBUG ] 22:37:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:37:34 DEBUG } 22:37:34 DEBUG }, 22:37:34 DEBUG ] 22:37:34 DEBUG "value": [ 22:37:34 DEBUG } 22:37:34 DEBUG 22:37:34 DEBUG } 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "0" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649054" 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "instance": "10.142.0.107:9100" 22:37:34 DEBUG "pod": "ds-idrepo-0" 22:37:34 DEBUG }, 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "instance": "10.142.0.43:9100" 22:37:34 DEBUG "pod": "ds-idrepo-2" 22:37:34 DEBUG }, 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "4144889377792" 22:37:34 DEBUG "0" 22:37:34 DEBUG ] 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO [loop_until]: OK (rc = 0) 22:37:34 DEBUG --- stdout --- 22:37:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22177m 8762Mi am-767f6b67b5-qcqz2 21938m 8879Mi am-767f6b67b5-vbltz 23784m 8936Mi ds-cts-0 20073m 12050Mi ds-cts-1 20663m 11712Mi ds-cts-2 20860m 11905Mi ds-idrepo-0 9437m 3907Mi ds-idrepo-1 9215m 3998Mi ds-idrepo-2 9444m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1179Mi idm-b85f9d978-wnhxb 4m 1239Mi lodemon-5497cb87d7-mlld4 1m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11692m 3344Mi 22:37:34 DEBUG --- stderr --- 22:37:34 DEBUG 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649054" 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "instance": "10.142.0.107:9100" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "instance": "10.142.0.43:9100" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "100378508.66666666" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "instance": "10.142.0.107:9100" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "instance": "10.142.0.43:9100" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "100377616.36666666" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649054" 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "3330.7666666666664" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG [...] 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "3331.7" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG { 22:37:34 DEBUG { 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG "data": { 22:37:34 DEBUG "data": { 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG "result": [ 22:37:34 INFO [http_cmd]: http status code OK 22:37:34 DEBUG "result": [ 22:37:34 DEBUG { 22:37:34 DEBUG --- status code --- 22:37:34 DEBUG { 22:37:34 DEBUG "metric": { 22:37:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:37:34 DEBUG --- http response --- 22:37:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:37:34 DEBUG }, 22:37:34 DEBUG }, 22:37:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:37:34 DEBUG "value": [ 22:37:34 DEBUG "value": [ 22:37:34 DEBUG ----- output ----- 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG { 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "status": "success", 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "data": { 22:37:34 DEBUG [...] 22:37:34 DEBUG [...] 22:37:34 DEBUG "resultType": "vector", 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "result": [ 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG { 22:37:34 DEBUG 22:37:34 DEBUG 22:37:34 DEBUG "metric": { 22:37:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:37:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:37:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:37:34 DEBUG }, 22:37:34 DEBUG }, 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG "value": [ 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG 22:37:34 DEBUG "11.714912685211223" 22:37:34 DEBUG "600976" 22:37:34 DEBUG 22:37:34 DEBUG ] 22:37:34 DEBUG ] 22:37:34 DEBUG 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 DEBUG [...] 22:37:34 DEBUG ] 22:37:34 DEBUG ] 22:37:34 DEBUG 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 DEBUG 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 DEBUG 22:37:34 DEBUG "pod": "ds-cts-1" 22:37:34 DEBUG }, 22:37:34 DEBUG "value": [ 22:37:34 DEBUG 1713649054, 22:37:34 DEBUG "68565101.27884758" 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG ] 22:37:34 DEBUG } 22:37:34 DEBUG } 22:37:34 INFO 22:37:34 INFO [loop_until]: kubectl --namespace=xlou top node 22:37:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:37:34 INFO [loop_until]: OK (rc = 0) 22:37:34 DEBUG --- stdout --- 22:37:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 6299Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9916m 33% 7664Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21826m 73% 11490Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3392Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20570m 68% 15275Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5203Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22249m 74% 11786Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19705m 66% 3134Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24699m 82% 11174Mi 10% gke-xlou-cdm-ds-0f601751-j488 10252m 34% 7286Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20280m 67% 16705Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 9994m 33% 7976Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20683m 69% 14568Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11879m 74% 11259Mi 19% 22:37:34 DEBUG --- stderr --- 22:37:34 DEBUG 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649114" 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649114" 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "pod": "ds-cts-0" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "ds-cts-2" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "0.3820333333333334" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "instance": "10.142.0.107:9100" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "instance": "10.142.0.43:9100" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "0.050666666666660386" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 ERROR [http_cmd]: ERROR 22:38:34 ERROR --- status code --- 22:38:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:38:34 ERROR --- http response --- 22:38:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:38:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "pod": "ds-idrepo-0" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "ds-idrepo-2" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "0" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "0" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "pod": "ds-idrepo-1" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "0" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649114" 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "3333.4" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "3603984384" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649114" 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG ----- output ----- 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 DEBUG { 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:38:34 DEBUG "result": [ 22:38:34 DEBUG "result": [ 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG { 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "pod": "ds-cts-0" 22:38:34 DEBUG "pod": "ds-idrepo-1" 22:38:34 DEBUG "data": { 22:38:34 DEBUG }, 22:38:34 DEBUG }, 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "value": [ 22:38:34 DEBUG "value": [ 22:38:34 DEBUG "result": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG { 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "metric": { 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "ds-idrepo-0" 22:38:34 DEBUG [...] 22:38:34 DEBUG [...] 22:38:34 DEBUG }, 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "ds-cts-2" 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:38:34 DEBUG 22:38:34 DEBUG }, 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG }, 22:38:34 DEBUG [...] 22:38:34 DEBUG "value": [ 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:38:34 DEBUG 22:38:34 DEBUG "0.03333333333333333" 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG "0.000020087397800980598" 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG 22:38:34 DEBUG ] 22:38:34 DEBUG { 22:38:34 DEBUG ] 22:38:34 DEBUG { 22:38:34 DEBUG "pod": "ds-cts-2" 22:38:34 DEBUG } 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG } 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG }, 22:38:34 DEBUG ] 22:38:34 DEBUG "data": { 22:38:34 DEBUG ] 22:38:34 DEBUG "data": { 22:38:34 DEBUG "value": [ 22:38:34 DEBUG } 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG } 22:38:34 DEBUG "resultType": "vector", 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG } 22:38:34 DEBUG "result": [ 22:38:34 DEBUG } 22:38:34 DEBUG "result": [ 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG "0" 22:38:34 DEBUG { 22:38:34 DEBUG { 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG ] 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG } 22:38:34 DEBUG "instance": "10.142.0.107:9100" 22:38:34 DEBUG "pod": "ds-idrepo-1" 22:38:34 DEBUG ] 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:38:34 DEBUG }, 22:38:34 DEBUG }, 22:38:34 DEBUG } 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG "value": [ 22:38:34 DEBUG "value": [ 22:38:34 DEBUG } 22:38:34 DEBUG { 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "data": { 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG [...] 22:38:34 DEBUG [...] 22:38:34 DEBUG "result": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG { 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "metric": { 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "ds-cts-0" 22:38:34 DEBUG "instance": "10.142.0.43:9100" 22:38:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:38:34 DEBUG }, 22:38:34 DEBUG }, 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG "value": [ 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG 22:38:34 DEBUG "0.17433333333333867" 22:38:34 DEBUG "0" 22:38:34 DEBUG 22:38:34 DEBUG ] 22:38:34 DEBUG ] 22:38:34 DEBUG [...] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 DEBUG 22:38:34 DEBUG ] 22:38:34 DEBUG ] 22:38:34 DEBUG 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 DEBUG 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 DEBUG "pod": "ds-cts-2" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "0" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "pod": "ds-idrepo-0" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "ds-idrepo-2" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "0" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649114" 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "101.875" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "3361.133333333333" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "instance": "10.142.0.107:9100" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "instance": "10.142.0.43:9100" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "100745323.3" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649114" 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO 22:38:34 INFO [loop_until]: kubectl --namespace=xlou top pods 22:38:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "pod": "ds-idrepo-0" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:34 DEBUG 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG "pod": "ds-idrepo-2" 22:38:34 DEBUG { 22:38:34 DEBUG }, 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "value": [ 22:38:34 DEBUG "data": { 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "0" 22:38:34 DEBUG "result": [ 22:38:34 DEBUG ] 22:38:34 DEBUG { 22:38:34 DEBUG } 22:38:34 DEBUG "metric": { 22:38:34 DEBUG ] 22:38:34 DEBUG "instance": "10.142.0.107:9100" 22:38:34 DEBUG } 22:38:34 DEBUG }, 22:38:34 DEBUG } 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "instance": "10.142.0.43:9100" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "4149039270912" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649114" 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "instance": "10.142.0.107:9100" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "instance": "10.142.0.43:9100" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "100746852.46666667" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649114" 22:38:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649114" 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "3361.9666666666667" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "3334.5" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG { 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG "pod": "ds-idrepo-1" 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG }, 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG "value": [ 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "0" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:38:34 INFO [http_cmd]: http status code OK 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG --- status code --- 22:38:34 DEBUG { 22:38:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG --- http response --- 22:38:34 DEBUG "data": { 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG "result": [ 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG ----- output ----- 22:38:34 DEBUG "11.666878463074674" 22:38:34 DEBUG { 22:38:34 DEBUG ] 22:38:34 DEBUG "status": "success", 22:38:34 DEBUG } 22:38:34 DEBUG "data": { 22:38:34 DEBUG ] 22:38:34 DEBUG "resultType": "vector", 22:38:34 DEBUG } 22:38:34 DEBUG "result": [ 22:38:34 DEBUG } 22:38:34 DEBUG { 22:38:34 DEBUG "metric": { 22:38:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG [...] 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG 22:38:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:38:34 DEBUG }, 22:38:34 DEBUG "value": [ 22:38:34 DEBUG 1713649114, 22:38:34 DEBUG "600074.5" 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG ] 22:38:34 DEBUG } 22:38:34 DEBUG } 22:38:34 INFO [loop_until]: OK (rc = 0) 22:38:34 DEBUG --- stdout --- 22:38:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22118m 8771Mi am-767f6b67b5-qcqz2 22051m 8884Mi am-767f6b67b5-vbltz 24635m 8943Mi ds-cts-0 20870m 12070Mi ds-cts-1 19878m 11932Mi ds-cts-2 20430m 12196Mi ds-idrepo-0 9598m 3907Mi ds-idrepo-1 9276m 3956Mi ds-idrepo-2 9454m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 4m 1239Mi lodemon-5497cb87d7-mlld4 1m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11402m 3347Mi 22:38:34 DEBUG --- stderr --- 22:38:34 DEBUG 22:38:34 INFO 22:38:34 INFO [loop_until]: kubectl --namespace=xlou top node 22:38:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:38:34 INFO [loop_until]: OK (rc = 0) 22:38:34 DEBUG --- stdout --- 22:38:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6310Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9933m 33% 7673Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21841m 73% 11513Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3381Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20295m 67% 15286Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 170m 0% 5220Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 21582m 72% 11789Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19824m 66% 3133Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24861m 83% 11179Mi 10% gke-xlou-cdm-ds-0f601751-j488 10240m 34% 7294Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20330m 68% 16944Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10036m 33% 7939Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20844m 69% 14919Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11929m 75% 11257Mi 19% 22:38:34 DEBUG --- stderr --- 22:38:34 DEBUG 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "0" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649174" 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:39:34 DEBUG "pod": "ds-cts-0" 22:39:34 DEBUG }, 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:39:34 DEBUG "pod": "ds-cts-2" 22:39:34 DEBUG }, 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "3356.5666666666666" 22:39:34 DEBUG "0.43260000000000004" 22:39:34 DEBUG ] 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 ERROR [http_cmd]: ERROR 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 ERROR --- status code --- 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 ERROR --- http response --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "instance": "10.142.0.107:9100" 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG "pod": "ds-idrepo-0" 22:39:34 DEBUG }, 22:39:34 DEBUG }, 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:34 DEBUG "value": [ 22:39:34 DEBUG "value": [ 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG { 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "data": { 22:39:34 DEBUG [...] 22:39:34 DEBUG [...] 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG "result": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG { 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "instance": "10.142.0.43:9100" 22:39:34 DEBUG "pod": "ds-cts-1" 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG "pod": "ds-idrepo-0" 22:39:34 DEBUG }, 22:39:34 DEBUG }, 22:39:34 DEBUG }, 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:34 DEBUG "value": [ 22:39:34 DEBUG "value": [ 22:39:34 DEBUG "value": [ 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG 22:39:34 DEBUG { 22:39:34 DEBUG "0.05100000000000288" 22:39:34 DEBUG "31151000.155680794" 22:39:34 DEBUG 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG ] 22:39:34 DEBUG ] 22:39:34 DEBUG 22:39:34 DEBUG "data": { 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 DEBUG [...] 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG ] 22:39:34 DEBUG ] 22:39:34 DEBUG 22:39:34 DEBUG "result": [ 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 DEBUG 22:39:34 DEBUG { 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 DEBUG 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "pod": "ds-idrepo-2" 22:39:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:39:34 DEBUG }, 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG 22:39:34 DEBUG "0" 22:39:34 DEBUG 22:39:34 DEBUG ] 22:39:34 DEBUG 22:39:34 DEBUG } 22:39:34 DEBUG [...] 22:39:34 DEBUG ] 22:39:34 DEBUG 22:39:34 DEBUG } 22:39:34 DEBUG 22:39:34 DEBUG } 22:39:34 DEBUG 22:39:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "3608965120" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "pod": "ds-idrepo-0" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "0" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "pod": "ds-cts-0" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "pod": "ds-cts-2" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "0" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649174" 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG { 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:34 DEBUG "data": { 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG "pod": "ds-idrepo-0" 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG }, 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG "value": [ 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "pod": "ds-idrepo-0" 22:39:34 DEBUG 22:39:34 DEBUG { 22:39:34 DEBUG "pod": "ds-cts-0" 22:39:34 DEBUG }, 22:39:34 DEBUG 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG [...] 22:39:34 DEBUG "data": { 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 DEBUG 22:39:34 DEBUG "result": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG { 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG "pod": "ds-cts-2" 22:39:34 DEBUG "metric": { 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG }, 22:39:34 DEBUG "instance": "10.142.0.107:9100" 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "value": [ 22:39:34 DEBUG }, 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:39:34 DEBUG "0" 22:39:34 DEBUG 22:39:34 DEBUG "pod": "ds-cts-2" 22:39:34 DEBUG }, 22:39:34 DEBUG ] 22:39:34 DEBUG 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG } 22:39:34 DEBUG 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG ] 22:39:34 DEBUG [...] 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "0" 22:39:34 DEBUG } 22:39:34 DEBUG 22:39:34 DEBUG "0" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG "instance": "10.142.0.43:9100" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG }, 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 DEBUG "value": [ 22:39:34 DEBUG } 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "0.17376666666671858" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "pod": "ds-idrepo-0" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "pod": "ds-idrepo-2" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "0" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649174" 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "101.875" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "3356.0333333333333" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649174" 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "instance": "10.142.0.107:9100" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "instance": "10.142.0.43:9100" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "101047377.36666667" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649174" 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "instance": "10.142.0.107:9100" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "instance": "10.142.0.43:9100" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "101047959.23333333" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO 22:39:34 INFO [loop_until]: kubectl --namespace=xlou top pods 22:39:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "11.676771453070856" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "pod": "ds-idrepo-0" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:39:34 DEBUG 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG 22:39:34 DEBUG { 22:39:34 DEBUG "pod": "ds-idrepo-2" 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG }, 22:39:34 DEBUG "data": { 22:39:34 DEBUG "value": [ 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "result": [ 22:39:34 DEBUG "0" 22:39:34 DEBUG { 22:39:34 DEBUG ] 22:39:34 DEBUG "metric": { 22:39:34 DEBUG } 22:39:34 DEBUG "instance": "10.142.0.107:9100" 22:39:34 DEBUG ] 22:39:34 DEBUG }, 22:39:34 DEBUG } 22:39:34 DEBUG "value": [ 22:39:34 DEBUG } 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "instance": "10.142.0.43:9100" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "4153213152256" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649174" 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "3356.8333333333335" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:39:34 DEBUG { 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG { 22:39:34 DEBUG "data": { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "result": [ 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG { 22:39:34 DEBUG "result": [ 22:39:34 DEBUG "metric": { 22:39:34 DEBUG { 22:39:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:39:34 DEBUG "metric": { 22:39:34 DEBUG }, 22:39:34 DEBUG "pod": "ds-idrepo-0" 22:39:34 DEBUG "value": [ 22:39:34 DEBUG }, 22:39:34 DEBUG 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:39:34 DEBUG 22:39:34 DEBUG }, 22:39:34 DEBUG "pod": "ds-cts-1" 22:39:34 DEBUG "value": [ 22:39:34 DEBUG }, 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG "3347.5666666666666" 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG ] 22:39:34 DEBUG "71601503.0046393" 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 INFO [http_cmd]: http status code OK 22:39:34 DEBUG --- status code --- 22:39:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:39:34 DEBUG --- http response --- 22:39:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:39:34 DEBUG ----- output ----- 22:39:34 DEBUG { 22:39:34 DEBUG "status": "success", 22:39:34 DEBUG "data": { 22:39:34 DEBUG "resultType": "vector", 22:39:34 DEBUG "result": [ 22:39:34 DEBUG { 22:39:34 DEBUG "metric": { 22:39:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG [...] 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG 22:39:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:39:34 DEBUG }, 22:39:34 DEBUG "value": [ 22:39:34 DEBUG 1713649174, 22:39:34 DEBUG "600369" 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG ] 22:39:34 DEBUG } 22:39:34 DEBUG } 22:39:34 INFO [loop_until]: OK (rc = 0) 22:39:34 DEBUG --- stdout --- 22:39:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21820m 8776Mi am-767f6b67b5-qcqz2 22731m 8902Mi am-767f6b67b5-vbltz 24020m 8933Mi ds-cts-0 20198m 12076Mi ds-cts-1 20649m 12017Mi ds-cts-2 21038m 12346Mi ds-idrepo-0 9748m 3907Mi ds-idrepo-1 9593m 3955Mi ds-idrepo-2 9606m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11536m 3352Mi 22:39:34 DEBUG --- stderr --- 22:39:34 DEBUG 22:39:34 INFO 22:39:34 INFO [loop_until]: kubectl --namespace=xlou top node 22:39:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:39:34 INFO [loop_until]: OK (rc = 0) 22:39:34 DEBUG --- stdout --- 22:39:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6295Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9945m 33% 7671Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22007m 73% 11511Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3383Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20189m 67% 15298Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5232Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22460m 75% 11809Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19823m 66% 3127Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24870m 83% 11180Mi 10% gke-xlou-cdm-ds-0f601751-j488 10286m 34% 7293Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20252m 67% 17004Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10092m 33% 7934Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20119m 67% 15040Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11561m 72% 11258Mi 19% 22:39:34 DEBUG --- stderr --- 22:39:34 DEBUG 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649234" 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "0" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "3613126656" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649234" 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 ERROR [http_cmd]: ERROR 22:40:34 ERROR --- status code --- 22:40:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:40:34 ERROR --- http response --- 22:40:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:40:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 DEBUG --- status code --- 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "pod": "ds-idrepo-0" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "pod": "ds-idrepo-2" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "0" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "0" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649234" 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "pod": "ds-cts-0" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "pod": "ds-cts-2" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "0.5057666666666665" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "3320.3333333333335" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "instance": "10.142.0.107:9100" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "instance": "10.142.0.43:9100" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "0.07633333333334916" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "pod": "ds-cts-0" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "pod": "ds-cts-2" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "0" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "pod": "ds-idrepo-0" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "pod": "ds-cts-2" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "0" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "0" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649234" 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649234" 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "pod": "ds-cts-0" 22:40:34 DEBUG "instance": "10.142.0.107:9100" 22:40:34 DEBUG }, 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG [...] 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649234" 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG "pod": "ds-cts-2" 22:40:34 DEBUG "instance": "10.142.0.43:9100" 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG }, 22:40:34 DEBUG }, 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG "value": [ 22:40:34 DEBUG "value": [ 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG "0" 22:40:34 DEBUG "0.32980000000004756" 22:40:34 DEBUG { 22:40:34 DEBUG { 22:40:34 DEBUG { 22:40:34 DEBUG ] 22:40:34 DEBUG ] 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 DEBUG "data": { 22:40:34 DEBUG "data": { 22:40:34 DEBUG "data": { 22:40:34 DEBUG ] 22:40:34 DEBUG ] 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 DEBUG "result": [ 22:40:34 DEBUG "result": [ 22:40:34 DEBUG "result": [ 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 DEBUG { 22:40:34 DEBUG { 22:40:34 DEBUG { 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:40:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:40:34 DEBUG "pod": "ds-idrepo-0" 22:40:34 DEBUG }, 22:40:34 DEBUG }, 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG "value": [ 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG [...] 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649234" 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:40:34 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG "pod": "ds-idrepo-2" 22:40:34 DEBUG }, 22:40:34 DEBUG }, 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG "value": [ 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "11.742755915012914" 22:40:34 DEBUG "0" 22:40:34 DEBUG "0" 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:34 DEBUG ] 22:40:34 DEBUG ] 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG ] 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG } 22:40:34 DEBUG { 22:40:34 DEBUG ] 22:40:34 DEBUG ] 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG ] 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG } 22:40:34 DEBUG "data": { 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "data": { 22:40:34 DEBUG "instance": "10.142.0.107:9100" 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649234" 22:40:34 DEBUG "resultType": "vector", 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG }, 22:40:34 DEBUG "result": [ 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG "value": [ 22:40:34 DEBUG { 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG 22:40:34 DEBUG "metric": { 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG 22:40:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:40:34 DEBUG 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:34 DEBUG }, 22:40:34 DEBUG [...] 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG { 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "data": { 22:40:34 DEBUG 22:40:34 DEBUG "instance": "10.142.0.43:9100" 22:40:34 DEBUG "resultType": "vector", 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG [...] 22:40:34 DEBUG }, 22:40:34 DEBUG "result": [ 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG 22:40:34 DEBUG "value": [ 22:40:34 DEBUG { 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "metric": { 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG 22:40:34 DEBUG "4157848329216" 22:40:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:40:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:34 DEBUG ] 22:40:34 DEBUG }, 22:40:34 DEBUG }, 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG } 22:40:34 DEBUG "value": [ 22:40:34 DEBUG "value": [ 22:40:34 DEBUG { 22:40:34 DEBUG ] 22:40:34 DEBUG 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG } 22:40:34 DEBUG 22:40:34 DEBUG "3333.9666666666667" 22:40:34 DEBUG "data": { 22:40:34 DEBUG } 22:40:34 DEBUG 22:40:34 DEBUG ] 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG [...] 22:40:34 DEBUG } 22:40:34 DEBUG "result": [ 22:40:34 DEBUG 22:40:34 DEBUG ] 22:40:34 DEBUG { 22:40:34 DEBUG 22:40:34 DEBUG } 22:40:34 DEBUG "metric": { 22:40:34 DEBUG 22:40:34 DEBUG } 22:40:34 DEBUG "instance": "10.142.0.107:9100" 22:40:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:40:34 DEBUG }, 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG 22:40:34 DEBUG "101.875" 22:40:34 DEBUG 22:40:34 DEBUG ] 22:40:34 DEBUG [...] 22:40:34 DEBUG } 22:40:34 DEBUG 22:40:34 DEBUG ] 22:40:34 DEBUG 22:40:34 DEBUG } 22:40:34 DEBUG 22:40:34 DEBUG } 22:40:34 DEBUG "instance": "10.142.0.43:9100" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "100810874.5" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "instance": "10.142.0.107:9100" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "instance": "10.142.0.43:9100" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "100812264.16666666" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "pod": "ds-idrepo-0" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "pod": "ds-idrepo-2" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "0" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [loop_until]: kubectl --namespace=xlou top pods 22:40:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649234" 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "3333.5666666666666" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "3337.3" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "475.61542034370643" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [http_cmd]: http status code OK 22:40:34 DEBUG --- status code --- 22:40:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:40:34 DEBUG --- http response --- 22:40:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:40:34 DEBUG ----- output ----- 22:40:34 DEBUG { 22:40:34 DEBUG "status": "success", 22:40:34 DEBUG "data": { 22:40:34 DEBUG "resultType": "vector", 22:40:34 DEBUG "result": [ 22:40:34 DEBUG { 22:40:34 DEBUG "metric": { 22:40:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG [...] 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG 22:40:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:40:34 DEBUG }, 22:40:34 DEBUG "value": [ 22:40:34 DEBUG 1713649234, 22:40:34 DEBUG "600721" 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG ] 22:40:34 DEBUG } 22:40:34 DEBUG } 22:40:34 INFO [loop_until]: OK (rc = 0) 22:40:34 DEBUG --- stdout --- 22:40:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22328m 8778Mi am-767f6b67b5-qcqz2 22268m 8899Mi am-767f6b67b5-vbltz 24424m 8937Mi ds-cts-0 20067m 12077Mi ds-cts-1 19688m 12167Mi ds-cts-2 21120m 12339Mi ds-idrepo-0 9581m 3907Mi ds-idrepo-1 9388m 3955Mi ds-idrepo-2 9923m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 4m 1239Mi lodemon-5497cb87d7-mlld4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11865m 3368Mi 22:40:34 DEBUG --- stderr --- 22:40:34 DEBUG 22:40:34 INFO 22:40:34 INFO [loop_until]: kubectl --namespace=xlou top node 22:40:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:40:34 INFO [loop_until]: OK (rc = 0) 22:40:34 DEBUG --- stdout --- 22:40:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 6301Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9978m 33% 7677Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21847m 73% 11511Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3367Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19902m 66% 15318Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 149m 0% 5209Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22373m 74% 11821Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19736m 66% 3125Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24643m 82% 11185Mi 10% gke-xlou-cdm-ds-0f601751-j488 10282m 34% 7287Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19993m 66% 17181Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10056m 33% 7935Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20451m 68% 15022Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11993m 75% 11270Mi 19% 22:40:34 DEBUG --- stderr --- 22:40:34 DEBUG 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "0" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "3631017984" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "pod": "ds-cts-0" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "pod": "ds-cts-2" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "0.5227999999999995" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649294" 22:41:34 ERROR [http_cmd]: ERROR 22:41:34 ERROR --- status code --- 22:41:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:41:34 ERROR --- http response --- 22:41:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:41:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "pod": "ds-idrepo-0" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "0" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "instance": "10.142.0.107:9100" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "instance": "10.142.0.43:9100" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "0.08166666666666061" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "pod": "ds-idrepo-0" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "pod": "ds-idrepo-2" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "0" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "3356.6" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "pod": "ds-cts-0" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "pod": "ds-cts-2" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "0" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:41:34 DEBUG } 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG } 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "pod": "ds-idrepo-0" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "pod": "ds-cts-2" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "0" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "pod": "ds-idrepo-0" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "pod": "ds-idrepo-2" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "0" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "pod": "ds-cts-0" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "pod": "ds-cts-2" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "0" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "instance": "10.142.0.107:9100" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "instance": "10.142.0.43:9100" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "0.4168666666666534" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG ----- output ----- 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG { 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG { 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:34 DEBUG { 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG "data": { 22:41:34 DEBUG "data": { 22:41:34 DEBUG { 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG "data": { 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG { 22:41:34 DEBUG "data": { 22:41:34 DEBUG { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG "result": [ 22:41:34 DEBUG "data": { 22:41:34 DEBUG "result": [ 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:34 DEBUG "data": { 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:41:34 DEBUG "result": [ 22:41:34 DEBUG "data": { 22:41:34 DEBUG { 22:41:34 DEBUG { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG { 22:41:34 DEBUG { 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "result": [ 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG { 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "result": [ 22:41:34 DEBUG "pod": "ds-idrepo-0" 22:41:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:41:34 DEBUG { 22:41:34 DEBUG "pod": "ds-idrepo-0" 22:41:34 DEBUG "data": { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "instance": "10.142.0.107:9100" 22:41:34 DEBUG { 22:41:34 DEBUG }, 22:41:34 DEBUG }, 22:41:34 DEBUG "metric": { 22:41:34 DEBUG }, 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "data": { 22:41:34 DEBUG }, 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "value": [ 22:41:34 DEBUG "value": [ 22:41:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:41:34 DEBUG "value": [ 22:41:34 DEBUG "result": [ 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "value": [ 22:41:34 DEBUG "pod": "ds-idrepo-0" 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG }, 22:41:34 DEBUG 22:41:34 DEBUG { 22:41:34 DEBUG "result": [ 22:41:34 DEBUG }, 22:41:34 DEBUG "result": [ 22:41:34 DEBUG 22:41:34 DEBUG }, 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG "metric": { 22:41:34 DEBUG { 22:41:34 DEBUG "value": [ 22:41:34 DEBUG { 22:41:34 DEBUG 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:41:34 DEBUG "metric": { 22:41:34 DEBUG 22:41:34 DEBUG "metric": { 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG }, 22:41:34 DEBUG "instance": "10.142.0.107:9100" 22:41:34 DEBUG 22:41:34 DEBUG "instance": "10.142.0.107:9100" 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "value": [ 22:41:34 DEBUG }, 22:41:34 DEBUG 22:41:34 DEBUG }, 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "value": [ 22:41:34 DEBUG [...] 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "pod": "ds-idrepo-2" 22:41:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:41:34 DEBUG 22:41:34 DEBUG "pod": "ds-idrepo-2" 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "instance": "10.142.0.43:9100" 22:41:34 DEBUG 22:41:34 DEBUG }, 22:41:34 DEBUG }, 22:41:34 DEBUG 22:41:34 DEBUG }, 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG }, 22:41:34 DEBUG 22:41:34 DEBUG "value": [ 22:41:34 DEBUG "value": [ 22:41:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:41:34 DEBUG [...] 22:41:34 DEBUG "value": [ 22:41:34 DEBUG "pod": "ds-idrepo-2" 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG }, 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG }, 22:41:34 DEBUG 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG }, 22:41:34 DEBUG "0" 22:41:34 DEBUG "11.799120258519691" 22:41:34 DEBUG "value": [ 22:41:34 DEBUG "0" 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG "4163280487424" 22:41:34 DEBUG "value": [ 22:41:34 DEBUG ] 22:41:34 DEBUG ] 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG ] 22:41:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:41:34 DEBUG 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG 22:41:34 DEBUG ] 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG "3332.866666666667" 22:41:34 DEBUG } 22:41:34 DEBUG }, 22:41:34 DEBUG "instance": "10.142.0.43:9100" 22:41:34 DEBUG "101.875" 22:41:34 DEBUG "instance": "10.142.0.43:9100" 22:41:34 DEBUG } 22:41:34 DEBUG "0" 22:41:34 DEBUG ] 22:41:34 DEBUG ] 22:41:34 DEBUG ] 22:41:34 DEBUG ] 22:41:34 DEBUG "value": [ 22:41:34 DEBUG }, 22:41:34 DEBUG ] 22:41:34 DEBUG }, 22:41:34 DEBUG ] 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG } 22:41:34 DEBUG "value": [ 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG "3331.7" 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "100389843" 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG ] 22:41:34 DEBUG ] 22:41:34 DEBUG "100391462.56666666" 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 INFO 22:41:34 INFO [loop_until]: kubectl --namespace=xlou top pods 22:41:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649294" 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "pod": "ds-idrepo-0" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "1313.2940563504467" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "3334.4666666666667" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 INFO [http_cmd]: http status code OK 22:41:34 DEBUG --- status code --- 22:41:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:41:34 DEBUG --- http response --- 22:41:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:41:34 DEBUG ----- output ----- 22:41:34 DEBUG { 22:41:34 DEBUG "status": "success", 22:41:34 DEBUG "data": { 22:41:34 DEBUG "resultType": "vector", 22:41:34 DEBUG "result": [ 22:41:34 DEBUG { 22:41:34 DEBUG "metric": { 22:41:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG [...] 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG 22:41:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:41:34 DEBUG }, 22:41:34 DEBUG "value": [ 22:41:34 DEBUG 1713649294, 22:41:34 DEBUG "601577.5" 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG ] 22:41:34 DEBUG } 22:41:34 DEBUG } 22:41:34 INFO [loop_until]: OK (rc = 0) 22:41:34 DEBUG --- stdout --- 22:41:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21121m 8774Mi am-767f6b67b5-qcqz2 21959m 8900Mi am-767f6b67b5-vbltz 24144m 8938Mi ds-cts-0 19955m 12073Mi ds-cts-1 20019m 12174Mi ds-cts-2 20295m 12361Mi ds-idrepo-0 9508m 3907Mi ds-idrepo-1 9324m 3955Mi ds-idrepo-2 9745m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 4m 1239Mi lodemon-5497cb87d7-mlld4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11792m 3369Mi 22:41:34 DEBUG --- stderr --- 22:41:34 DEBUG 22:41:34 INFO 22:41:34 INFO [loop_until]: kubectl --namespace=xlou top node 22:41:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:41:34 INFO [loop_until]: OK (rc = 0) 22:41:34 DEBUG --- stdout --- 22:41:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6295Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9984m 33% 7680Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21184m 70% 11514Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3387Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19358m 64% 15305Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 164m 0% 5209Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22336m 74% 11811Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19764m 66% 3121Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24691m 82% 11183Mi 10% gke-xlou-cdm-ds-0f601751-j488 10279m 34% 7294Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19778m 66% 17165Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10075m 33% 7933Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20528m 68% 15061Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11920m 75% 11274Mi 19% 22:41:34 DEBUG --- stderr --- 22:41:34 DEBUG 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649354" 22:42:34 ERROR [http_cmd]: ERROR 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 ERROR --- status code --- 22:42:34 DEBUG --- status code --- 22:42:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 ERROR --- http response --- 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:42:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:42:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG "3632480256" 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "instance": "10.142.0.107:9100" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "instance": "10.142.0.43:9100" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG "0.08833333333332878" 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG "0" 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "pod": "ds-cts-0" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "ds-cts-2" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG "0.5091000000000003" 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG "0" 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "pod": "ds-idrepo-0" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "ds-idrepo-2" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG "0" 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG "3354" 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "data": { 22:42:34 DEBUG "pod": "ds-cts-0" 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG }, 22:42:34 DEBUG "result": [ 22:42:34 DEBUG "value": [ 22:42:34 DEBUG { 22:42:34 DEBUG 22:42:34 DEBUG "metric": { 22:42:34 DEBUG 22:42:34 DEBUG "pod": "ds-idrepo-0" 22:42:34 DEBUG 22:42:34 DEBUG }, 22:42:34 DEBUG [...] 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "ds-cts-2" 22:42:34 DEBUG [...] 22:42:34 DEBUG }, 22:42:34 DEBUG 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG 22:42:34 DEBUG "0" 22:42:34 DEBUG "pod": "ds-cts-2" 22:42:34 DEBUG ] 22:42:34 DEBUG }, 22:42:34 DEBUG } 22:42:34 DEBUG "value": [ 22:42:34 DEBUG ] 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG } 22:42:34 DEBUG "0" 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG "306414826.8593006" 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "pod": "ds-cts-0" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "ds-cts-2" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG "0" 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG "3333.9666666666667" 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG { 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:34 DEBUG "result": [ 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:42:34 DEBUG "data": { 22:42:34 DEBUG }, 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "value": [ 22:42:34 DEBUG "result": [ 22:42:34 DEBUG 22:42:34 DEBUG { 22:42:34 DEBUG 22:42:34 DEBUG "metric": { 22:42:34 DEBUG 22:42:34 DEBUG "instance": "10.142.0.107:9100" 22:42:34 DEBUG [...] 22:42:34 DEBUG }, 22:42:34 DEBUG 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:42:34 DEBUG 22:42:34 DEBUG }, 22:42:34 DEBUG [...] 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG 22:42:34 DEBUG "101.875" 22:42:34 DEBUG 22:42:34 DEBUG ] 22:42:34 DEBUG "instance": "10.142.0.43:9100" 22:42:34 DEBUG } 22:42:34 DEBUG }, 22:42:34 DEBUG ] 22:42:34 DEBUG "value": [ 22:42:34 DEBUG } 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG } 22:42:34 DEBUG "0.42580000000007484" 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649354" 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:34 DEBUG --- status code --- 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:34 DEBUG { 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:34 DEBUG { 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG { 22:42:34 DEBUG "data": { 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG { 22:42:34 DEBUG "data": { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "data": { 22:42:34 DEBUG "result": [ 22:42:34 DEBUG "data": { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG { 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG "data": { 22:42:34 DEBUG "result": [ 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:42:34 DEBUG { 22:42:34 DEBUG "result": [ 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG "data": { 22:42:34 DEBUG "result": [ 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG "metric": { 22:42:34 DEBUG { 22:42:34 DEBUG "pod": "ds-idrepo-0" 22:42:34 DEBUG { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "data": { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG { 22:42:34 DEBUG { 22:42:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:42:34 DEBUG "metric": { 22:42:34 DEBUG }, 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "result": [ 22:42:34 DEBUG "result": [ 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG }, 22:42:34 DEBUG "instance": "10.142.0.107:9100" 22:42:34 DEBUG "value": [ 22:42:34 DEBUG "pod": "ds-idrepo-0" 22:42:34 DEBUG }, 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG { 22:42:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:42:34 DEBUG "data": { 22:42:34 DEBUG "value": [ 22:42:34 DEBUG }, 22:42:34 DEBUG 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG }, 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "instance": "10.142.0.107:9100" 22:42:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:42:34 DEBUG "value": [ 22:42:34 DEBUG "result": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:42:34 DEBUG }, 22:42:34 DEBUG }, 22:42:34 DEBUG 22:42:34 DEBUG { 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG "metric": { 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "instance": "10.142.0.107:9100" 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG }, 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "ds-idrepo-2" 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:42:34 DEBUG 22:42:34 DEBUG }, 22:42:34 DEBUG 22:42:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG }, 22:42:34 DEBUG "instance": "10.142.0.43:9100" 22:42:34 DEBUG "value": [ 22:42:34 DEBUG "pod": "ds-idrepo-2" 22:42:34 DEBUG }, 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:42:34 DEBUG 22:42:34 DEBUG "value": [ 22:42:34 DEBUG }, 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG }, 22:42:34 DEBUG [...] 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG "0" 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG 22:42:34 DEBUG "instance": "10.142.0.43:9100" 22:42:34 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG "3334.733333333333" 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG ] 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG "11.729777505972807" 22:42:34 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:42:34 DEBUG }, 22:42:34 DEBUG }, 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG 22:42:34 DEBUG ] 22:42:34 DEBUG "100332293.2" 22:42:34 DEBUG } 22:42:34 DEBUG "0" 22:42:34 DEBUG ] 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG "value": [ 22:42:34 DEBUG "600780" 22:42:34 DEBUG 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG ] 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG ] 22:42:34 DEBUG "instance": "10.142.0.43:9100" 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG "4168448963584" 22:42:34 DEBUG "22.211894467246736" 22:42:34 DEBUG } 22:42:34 DEBUG }, 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG "537.9740600886554" 22:42:34 DEBUG ] 22:42:34 DEBUG ] 22:42:34 DEBUG ] 22:42:34 DEBUG "value": [ 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG "100332613.06666666" 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649354" 22:42:34 INFO [http_cmd]: http status code OK 22:42:34 DEBUG --- status code --- 22:42:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:42:34 DEBUG --- http response --- 22:42:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:42:34 DEBUG ----- output ----- 22:42:34 DEBUG { 22:42:34 DEBUG "status": "success", 22:42:34 DEBUG "data": { 22:42:34 DEBUG "resultType": "vector", 22:42:34 DEBUG "result": [ 22:42:34 DEBUG { 22:42:34 DEBUG "metric": { 22:42:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG [...] 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG 22:42:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:42:34 DEBUG }, 22:42:34 DEBUG "value": [ 22:42:34 DEBUG 1713649354, 22:42:34 DEBUG "3328.8333333333335" 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG ] 22:42:34 DEBUG } 22:42:34 DEBUG } 22:42:34 INFO 22:42:34 INFO [loop_until]: kubectl --namespace=xlou top pods 22:42:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:42:34 INFO [loop_until]: OK (rc = 0) 22:42:34 DEBUG --- stdout --- 22:42:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21867m 8775Mi am-767f6b67b5-qcqz2 22420m 8898Mi am-767f6b67b5-vbltz 25006m 8941Mi ds-cts-0 20591m 12083Mi ds-cts-1 19664m 12165Mi ds-cts-2 20349m 12356Mi ds-idrepo-0 9253m 3907Mi ds-idrepo-1 9252m 3956Mi ds-idrepo-2 9625m 3869Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11646m 3373Mi 22:42:34 DEBUG --- stderr --- 22:42:34 DEBUG 22:42:34 INFO 22:42:34 INFO [loop_until]: kubectl --namespace=xlou top node 22:42:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:42:35 INFO [loop_until]: OK (rc = 0) 22:42:35 DEBUG --- stdout --- 22:42:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6290Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9930m 33% 7670Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21878m 73% 11511Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3387Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19831m 66% 15320Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5213Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22372m 74% 11799Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19783m 66% 3125Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24910m 83% 11195Mi 10% gke-xlou-cdm-ds-0f601751-j488 10256m 34% 7284Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19056m 63% 17173Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10085m 33% 7944Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20668m 69% 15064Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11910m 74% 11273Mi 19% 22:42:35 DEBUG --- stderr --- 22:42:35 DEBUG 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649414" 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649414" 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649414" 22:43:34 ERROR [http_cmd]: ERROR 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 ERROR --- status code --- 22:43:34 DEBUG --- status code --- 22:43:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 ERROR --- http response --- 22:43:34 DEBUG --- http response --- 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:43:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:43:34 DEBUG --- status code --- 22:43:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "3634184192" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:43:34 DEBUG "instance": "10.142.0.107:9100" 22:43:34 DEBUG }, 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:43:34 DEBUG "instance": "10.142.0.43:9100" 22:43:34 DEBUG }, 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "0" 22:43:34 DEBUG "0.07833333333333786" 22:43:34 DEBUG ] 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "ds-cts-0" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "ds-cts-2" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "0.5493999999999999" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "ds-idrepo-0" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "ds-cts-2" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "0" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "overseer-0-967d957f7-zk66v" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "0" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "overseer-0-967d957f7-zk66v" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "208816.28860394566" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "ds-idrepo-0" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "ds-idrepo-2" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "0" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "3336.1666666666665" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "ds-cts-0" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "ds-cts-2" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "0" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "ds-cts-0" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "ds-cts-2" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "0" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "3302.633333333333" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649414" 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 DEBUG }, 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG "value": [ 22:43:34 DEBUG { 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "101.875" 22:43:34 DEBUG "data": { 22:43:34 DEBUG ] 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG } 22:43:34 DEBUG "result": [ 22:43:34 DEBUG ] 22:43:34 DEBUG { 22:43:34 DEBUG } 22:43:34 DEBUG "metric": { 22:43:34 DEBUG } 22:43:34 DEBUG "instance": "10.142.0.107:9100" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "instance": "10.142.0.43:9100" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "0.4073999999999775" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649414" 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649414" 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "ds-idrepo-0" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "ds-idrepo-2" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "0" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:34 DEBUG { 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG { 22:43:34 DEBUG "data": { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "result": [ 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG { 22:43:34 DEBUG "result": [ 22:43:34 DEBUG "metric": { 22:43:34 DEBUG { 22:43:34 DEBUG "instance": "10.142.0.107:9100" 22:43:34 DEBUG "metric": { 22:43:34 DEBUG }, 22:43:34 DEBUG "instance": "10.142.0.107:9100" 22:43:34 DEBUG "value": [ 22:43:34 DEBUG }, 22:43:34 DEBUG 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "instance": "10.142.0.43:9100" 22:43:34 DEBUG 22:43:34 DEBUG }, 22:43:34 DEBUG "instance": "10.142.0.43:9100" 22:43:34 DEBUG "value": [ 22:43:34 DEBUG }, 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "value": [ 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 DEBUG "4173734736896" 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "100357877.23333333" 22:43:34 DEBUG ] 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "overseer-0-967d957f7-zk66v" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "0" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649414" 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG { 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "ds-idrepo-0" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "ds-idrepo-2" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "0" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "instance": "10.142.0.107:9100" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "instance": "10.142.0.43:9100" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "100357789.13333333" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649414" 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "3299.5333333333333" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG { 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG "data": { 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG "result": [ 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG { 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG }, 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG "value": [ 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:43:34 DEBUG 22:43:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:43:34 DEBUG }, 22:43:34 DEBUG [...] 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG }, 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "11.715878741622276" 22:43:34 DEBUG 22:43:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:43:34 DEBUG ] 22:43:34 DEBUG "pod": "overseer-0-967d957f7-zk66v" 22:43:34 DEBUG }, 22:43:34 DEBUG } 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG ] 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG } 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "599911.5" 22:43:34 DEBUG } 22:43:34 DEBUG "23.182004032609353" 22:43:34 DEBUG ] 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649414" 22:43:34 INFO [http_cmd]: http status code OK 22:43:34 DEBUG --- status code --- 22:43:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:43:34 DEBUG --- http response --- 22:43:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:43:34 DEBUG ----- output ----- 22:43:34 DEBUG { 22:43:34 DEBUG "status": "success", 22:43:34 DEBUG "data": { 22:43:34 DEBUG "resultType": "vector", 22:43:34 DEBUG "result": [ 22:43:34 DEBUG { 22:43:34 DEBUG "metric": { 22:43:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG [...] 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG 22:43:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:43:34 DEBUG }, 22:43:34 DEBUG "value": [ 22:43:34 DEBUG 1713649414, 22:43:34 DEBUG "3319.733333333333" 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG ] 22:43:34 DEBUG } 22:43:34 DEBUG } 22:43:34 INFO 22:43:34 INFO [loop_until]: kubectl --namespace=xlou top pods 22:43:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:34 INFO [loop_until]: OK (rc = 0) 22:43:34 DEBUG --- stdout --- 22:43:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22060m 8776Mi am-767f6b67b5-qcqz2 22119m 8899Mi am-767f6b67b5-vbltz 24840m 8943Mi ds-cts-0 19897m 12155Mi ds-cts-1 20378m 12233Mi ds-cts-2 20274m 12489Mi ds-idrepo-0 9589m 3907Mi ds-idrepo-1 9156m 3955Mi ds-idrepo-2 9867m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-5497cb87d7-mlld4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11787m 3376Mi 22:43:34 DEBUG --- stderr --- 22:43:34 DEBUG 22:43:35 INFO 22:43:35 INFO [loop_until]: kubectl --namespace=xlou top node 22:43:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:43:35 INFO [loop_until]: OK (rc = 0) 22:43:35 DEBUG --- stdout --- 22:43:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6290Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9908m 33% 7663Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21790m 72% 11517Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3375Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19621m 65% 15349Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5199Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22301m 74% 11810Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19550m 65% 3123Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24651m 82% 11190Mi 10% gke-xlou-cdm-ds-0f601751-j488 10205m 34% 7291Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19982m 66% 17258Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 9969m 33% 7935Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20308m 68% 15196Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11877m 74% 11271Mi 19% 22:43:35 DEBUG --- stderr --- 22:43:35 DEBUG 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649474" 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "3639992320" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649474" 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "0" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "overseer-0-967d957f7-zk66v" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "79254412.15630488" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 ERROR [http_cmd]: ERROR 22:44:34 DEBUG --- status code --- 22:44:34 ERROR --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 ERROR --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "ds-idrepo-0" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "ds-cts-2" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:44:34 DEBUG "0" 22:44:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649474" 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "ds-idrepo-0" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "ds-idrepo-2" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "0" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "instance": "10.142.0.107:9100" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "instance": "10.142.0.43:9100" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "0.07166666666666213" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "ds-cts-0" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "ds-cts-2" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "0.48629999999999995" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "overseer-0-967d957f7-zk66v" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "0" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "3336.4666666666667" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "ds-cts-0" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "ds-cts-2" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "0" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "3347.0666666666666" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "ds-cts-0" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "ds-cts-2" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "0" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "ds-idrepo-0" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "ds-idrepo-2" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "0" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "overseer-0-967d957f7-zk66v" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "0" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649474" 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "101.875" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "instance": "10.142.0.107:9100" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "instance": "10.142.0.43:9100" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "0.38689999999994407" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649474" 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649474" 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649474" 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "11.65275522636024" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG "result": [ 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG { 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "instance": "10.142.0.107:9100" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:34 DEBUG 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG 22:44:34 DEBUG { 22:44:34 DEBUG [...] 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG 22:44:34 DEBUG "data": { 22:44:34 DEBUG 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG 22:44:34 DEBUG "result": [ 22:44:34 DEBUG "instance": "10.142.0.43:9100" 22:44:34 DEBUG { 22:44:34 DEBUG }, 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "value": [ 22:44:34 DEBUG "instance": "10.142.0.107:9100" 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG }, 22:44:34 DEBUG "4178610541568" 22:44:34 DEBUG "value": [ 22:44:34 DEBUG ] 22:44:34 DEBUG 22:44:34 DEBUG } 22:44:34 DEBUG 22:44:34 DEBUG ] 22:44:34 DEBUG 22:44:34 DEBUG } 22:44:34 DEBUG [...] 22:44:34 DEBUG } 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "instance": "10.142.0.43:9100" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "100324137.63333333" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649474" 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "ds-idrepo-0" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "ds-idrepo-2" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "0" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "instance": "10.142.0.107:9100" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "instance": "10.142.0.43:9100" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "100325738.19999999" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "3347.133333333333" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "599650.5" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "overseer-0-967d957f7-zk66v" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "23.35426803588245" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649474" 22:44:34 INFO [http_cmd]: http status code OK 22:44:34 DEBUG --- status code --- 22:44:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:44:34 DEBUG --- http response --- 22:44:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:44:34 DEBUG ----- output ----- 22:44:34 DEBUG { 22:44:34 DEBUG "status": "success", 22:44:34 DEBUG "data": { 22:44:34 DEBUG "resultType": "vector", 22:44:34 DEBUG "result": [ 22:44:34 DEBUG { 22:44:34 DEBUG "metric": { 22:44:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG [...] 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG 22:44:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:44:34 DEBUG }, 22:44:34 DEBUG "value": [ 22:44:34 DEBUG 1713649474, 22:44:34 DEBUG "3346.766666666667" 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG ] 22:44:34 DEBUG } 22:44:34 DEBUG } 22:44:34 INFO 22:44:34 INFO [loop_until]: kubectl --namespace=xlou top pods 22:44:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:44:35 INFO [loop_until]: OK (rc = 0) 22:44:35 DEBUG --- stdout --- 22:44:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22003m 8778Mi am-767f6b67b5-qcqz2 21989m 8900Mi am-767f6b67b5-vbltz 25024m 8956Mi ds-cts-0 19908m 12235Mi ds-cts-1 19239m 12241Mi ds-cts-2 20472m 12494Mi ds-idrepo-0 9457m 3907Mi ds-idrepo-1 9590m 3955Mi ds-idrepo-2 9490m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 1m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 12008m 3378Mi 22:44:35 DEBUG --- stderr --- 22:44:35 DEBUG 22:44:35 INFO 22:44:35 INFO [loop_until]: kubectl --namespace=xlou top node 22:44:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:44:35 INFO [loop_until]: OK (rc = 0) 22:44:35 DEBUG --- stdout --- 22:44:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6277Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9998m 33% 7678Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21880m 73% 11508Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3376Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20094m 67% 15475Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 168m 0% 5206Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 21650m 72% 11811Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19817m 66% 3132Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24837m 83% 11210Mi 10% gke-xlou-cdm-ds-0f601751-j488 10296m 34% 7287Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20132m 67% 17378Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 9748m 32% 7933Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20598m 68% 15203Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11958m 75% 11272Mi 19% 22:44:35 DEBUG --- stderr --- 22:44:35 DEBUG 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649534" 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649534" 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "pod": "ds-cts-0" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "ds-cts-2" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "0.5198000000000003" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "0" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "pod": "ds-idrepo-0" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "ds-cts-2" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "0" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 ERROR [http_cmd]: ERROR 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 ERROR --- status code --- 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG --- status code --- 22:45:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 ERROR --- http response --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649534" 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:45:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG { 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "data": { 22:45:34 DEBUG "data": { 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG "result": [ 22:45:34 DEBUG "result": [ 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG { 22:45:34 DEBUG { 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "data": { 22:45:34 DEBUG "pod": "ds-idrepo-0" 22:45:34 DEBUG "pod": "ds-idrepo-1" 22:45:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG }, 22:45:34 DEBUG }, 22:45:34 DEBUG }, 22:45:34 DEBUG "result": [ 22:45:34 DEBUG "value": [ 22:45:34 DEBUG "value": [ 22:45:34 DEBUG "value": [ 22:45:34 DEBUG { 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "metric": { 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "ds-idrepo-1" 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG }, 22:45:34 DEBUG [...] 22:45:34 DEBUG [...] 22:45:34 DEBUG [...] 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "ds-idrepo-2" 22:45:34 DEBUG "pod": "ds-cts-1" 22:45:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:45:34 DEBUG [...] 22:45:34 DEBUG }, 22:45:34 DEBUG }, 22:45:34 DEBUG }, 22:45:34 DEBUG 22:45:34 DEBUG "value": [ 22:45:34 DEBUG "value": [ 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG 22:45:34 DEBUG "0" 22:45:34 DEBUG "57422786.86236308" 22:45:34 DEBUG "3642855424" 22:45:34 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:45:34 DEBUG ] 22:45:34 DEBUG ] 22:45:34 DEBUG ] 22:45:34 DEBUG }, 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 DEBUG "value": [ 22:45:34 DEBUG ] 22:45:34 DEBUG ] 22:45:34 DEBUG ] 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG "0" 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG ] 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG } 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "instance": "10.142.0.107:9100" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "instance": "10.142.0.43:9100" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "0.07866666666665574" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "3339.1" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "pod": "ds-cts-0" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "ds-cts-2" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "0" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "pod": "ds-cts-0" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "ds-cts-2" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "0" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "3339.5333333333333" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649534" 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "instance": "10.142.0.107:9100" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "instance": "10.142.0.43:9100" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "0.4071666666666099" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "pod": "ds-idrepo-0" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "ds-idrepo-2" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "0" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "pod": "ds-idrepo-1" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "ds-cts-1" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "55134859.25844338" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "101.875" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649534" 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649534" 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649534" 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "11.625966021253317" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG --- status code --- 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG { 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG "data": { 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG { 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG }, 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "value": [ 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "instance": "10.142.0.107:9100" 22:45:34 DEBUG 22:45:34 DEBUG "instance": "10.142.0.107:9100" 22:45:34 DEBUG }, 22:45:34 DEBUG 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG }, 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG "instance": "10.142.0.43:9100" 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "instance": "10.142.0.43:9100" 22:45:34 DEBUG }, 22:45:34 DEBUG "3335.866666666667" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG ] 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG } 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "4183458495488" 22:45:34 DEBUG ] 22:45:34 DEBUG "100509858.93333334" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "599778" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649534" 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "pod": "ds-idrepo-0" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "ds-idrepo-2" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "0" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "instance": "10.142.0.107:9100" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "instance": "10.142.0.43:9100" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "100510926.13333333" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649534" 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "3343.7666666666664" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:34 INFO [http_cmd]: http status code OK 22:45:34 DEBUG --- status code --- 22:45:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:45:34 DEBUG --- http response --- 22:45:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:45:34 DEBUG ----- output ----- 22:45:34 DEBUG { 22:45:34 DEBUG "status": "success", 22:45:34 DEBUG "data": { 22:45:34 DEBUG "resultType": "vector", 22:45:34 DEBUG "result": [ 22:45:34 DEBUG { 22:45:34 DEBUG "metric": { 22:45:34 DEBUG "pod": "ds-idrepo-1" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG [...] 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG 22:45:34 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:45:34 DEBUG }, 22:45:34 DEBUG "value": [ 22:45:34 DEBUG 1713649534, 22:45:34 DEBUG "0" 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG ] 22:45:34 DEBUG } 22:45:34 DEBUG } 22:45:35 INFO 22:45:35 INFO [loop_until]: kubectl --namespace=xlou top pods 22:45:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:45:35 INFO [loop_until]: OK (rc = 0) 22:45:35 DEBUG --- stdout --- 22:45:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21953m 8774Mi am-767f6b67b5-qcqz2 22357m 8901Mi am-767f6b67b5-vbltz 24228m 8938Mi ds-cts-0 20413m 12274Mi ds-cts-1 19932m 12358Mi ds-cts-2 20498m 12486Mi ds-idrepo-0 9342m 3907Mi ds-idrepo-1 9318m 3955Mi ds-idrepo-2 9899m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-5497cb87d7-mlld4 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11498m 3380Mi 22:45:35 DEBUG --- stderr --- 22:45:35 DEBUG 22:45:35 INFO 22:45:35 INFO [loop_until]: kubectl --namespace=xlou top node 22:45:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:45:35 INFO [loop_until]: OK (rc = 0) 22:45:35 DEBUG --- stdout --- 22:45:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6288Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9638m 32% 7675Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21925m 73% 11505Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3376Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19910m 66% 15508Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5208Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22398m 75% 11817Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19864m 66% 3116Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24745m 82% 11200Mi 10% gke-xlou-cdm-ds-0f601751-j488 10223m 34% 7291Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 20121m 67% 17388Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10046m 33% 7938Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20437m 68% 15189Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11949m 75% 11271Mi 19% 22:45:35 DEBUG --- stderr --- 22:45:35 DEBUG 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "pod": "ds-idrepo-0" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "pod": "ds-cts-2" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "0" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "0" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 ERROR [http_cmd]: ERROR 22:46:34 ERROR --- status code --- 22:46:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:46:34 ERROR --- http response --- 22:46:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:46:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "instance": "10.142.0.107:9100" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "instance": "10.142.0.43:9100" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "0.10499999999999735" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "pod": "ds-idrepo-1" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "pod": "ds-idrepo-2" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "0" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "pod": "ds-cts-0" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "pod": "ds-cts-2" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "0.537500000000001" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG { 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "pod": "ds-idrepo-0" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:34 DEBUG 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG { 22:46:34 DEBUG [...] 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG "data": { 22:46:34 DEBUG 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:34 DEBUG 22:46:34 DEBUG "result": [ 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG "pod": "ds-idrepo-2" 22:46:34 DEBUG { 22:46:34 DEBUG { 22:46:34 DEBUG }, 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "value": [ 22:46:34 DEBUG "pod": "ds-idrepo-1" 22:46:34 DEBUG "data": { 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG }, 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "0" 22:46:34 DEBUG "value": [ 22:46:34 DEBUG "result": [ 22:46:34 DEBUG ] 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG 22:46:34 DEBUG { 22:46:34 DEBUG } 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG 22:46:34 DEBUG "metric": { 22:46:34 DEBUG ] 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG 22:46:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:46:34 DEBUG } 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG [...] 22:46:34 DEBUG }, 22:46:34 DEBUG } 22:46:34 DEBUG 22:46:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "pod": "ds-idrepo-2" 22:46:34 DEBUG 22:46:34 DEBUG }, 22:46:34 DEBUG [...] 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG 22:46:34 DEBUG "178198532.0829949" 22:46:34 DEBUG 22:46:34 DEBUG ] 22:46:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:46:34 DEBUG } 22:46:34 DEBUG }, 22:46:34 DEBUG ] 22:46:34 DEBUG "value": [ 22:46:34 DEBUG } 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG } 22:46:34 DEBUG "3642458112" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "3301.5666666666666" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "pod": "ds-cts-0" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "pod": "ds-cts-2" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "0" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:46:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG }, 22:46:34 DEBUG { 22:46:34 DEBUG "value": [ 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG 22:46:34 DEBUG "data": { 22:46:34 DEBUG 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG 22:46:34 DEBUG "result": [ 22:46:34 DEBUG [...] 22:46:34 DEBUG { 22:46:34 DEBUG 22:46:34 DEBUG "metric": { 22:46:34 DEBUG 22:46:34 DEBUG "pod": "ds-cts-0" 22:46:34 DEBUG 22:46:34 DEBUG }, 22:46:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:46:34 DEBUG "value": [ 22:46:34 DEBUG }, 22:46:34 DEBUG 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG 22:46:34 DEBUG "3336.2" 22:46:34 DEBUG [...] 22:46:34 DEBUG ] 22:46:34 DEBUG 22:46:34 DEBUG } 22:46:34 DEBUG 22:46:34 DEBUG ] 22:46:34 DEBUG 22:46:34 DEBUG } 22:46:34 DEBUG "pod": "ds-cts-2" 22:46:34 DEBUG } 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "10.833333333333334" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "pod": "ds-idrepo-0" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "pod": "ds-idrepo-2" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "0" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "instance": "10.142.0.107:9100" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "instance": "10.142.0.43:9100" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "0.4463333333333878" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "pod": "ds-idrepo-1" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "pod": "ds-idrepo-2" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "4082.9949051170347" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649594" 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG { 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:34 DEBUG "metric": { 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:46:34 DEBUG { 22:46:34 DEBUG }, 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "value": [ 22:46:34 DEBUG "data": { 22:46:34 DEBUG 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG 22:46:34 DEBUG "result": [ 22:46:34 DEBUG 22:46:34 DEBUG { 22:46:34 DEBUG [...] 22:46:34 DEBUG "metric": { 22:46:34 DEBUG 22:46:34 DEBUG "instance": "10.142.0.107:9100" 22:46:34 DEBUG 22:46:34 DEBUG }, 22:46:34 DEBUG 22:46:34 DEBUG "value": [ 22:46:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:46:34 DEBUG 22:46:34 DEBUG }, 22:46:34 DEBUG 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG [...] 22:46:34 DEBUG "101.875" 22:46:34 DEBUG 22:46:34 DEBUG ] 22:46:34 DEBUG 22:46:34 DEBUG } 22:46:34 DEBUG 22:46:34 DEBUG ] 22:46:34 DEBUG "instance": "10.142.0.43:9100" 22:46:34 DEBUG } 22:46:34 DEBUG }, 22:46:34 DEBUG } 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "100509606.9" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "11.782242090860883" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649594" 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "3336.3" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649594" 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "data": { 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "instance": "10.142.0.107:9100" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "instance": "10.142.0.43:9100" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "4188613534720" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG --- http response --- 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG { 22:46:34 DEBUG { 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG "status": "success", 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG "data": { 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:34 DEBUG "data": { 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG "result": [ 22:46:34 DEBUG { 22:46:34 DEBUG "result": [ 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG { 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:46:34 DEBUG "data": { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:46:34 DEBUG }, 22:46:34 DEBUG { 22:46:34 DEBUG "result": [ 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG { 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG "data": { 22:46:34 DEBUG "metric": { 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "pod": "ds-idrepo-1" 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "result": [ 22:46:34 DEBUG }, 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG { 22:46:34 DEBUG "value": [ 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG "metric": { 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "pod": "ds-idrepo-0" 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG }, 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:46:34 DEBUG "value": [ 22:46:34 DEBUG [...] 22:46:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:46:34 DEBUG }, 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 INFO [http_cmd]: http status code OK 22:46:34 DEBUG 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG 22:46:34 DEBUG --- status code --- 22:46:34 DEBUG 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "600353" 22:46:34 DEBUG [...] 22:46:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:46:34 DEBUG "pod": "ds-idrepo-2" 22:46:34 DEBUG "3334.9666666666667" 22:46:34 DEBUG ] 22:46:34 DEBUG 22:46:34 DEBUG --- http response --- 22:46:34 DEBUG }, 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG 22:46:34 DEBUG "value": [ 22:46:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG ----- output ----- 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG "pod": "ds-idrepo-2" 22:46:34 DEBUG "9.631898582292507" 22:46:34 DEBUG { 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:34 DEBUG }, 22:46:34 DEBUG ] 22:46:34 DEBUG "status": "success", 22:46:34 DEBUG } 22:46:34 DEBUG "value": [ 22:46:34 DEBUG } 22:46:34 DEBUG "data": { 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG ] 22:46:34 DEBUG "resultType": "vector", 22:46:34 DEBUG "0" 22:46:34 DEBUG } 22:46:34 DEBUG "result": [ 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG { 22:46:34 DEBUG } 22:46:34 DEBUG "metric": { 22:46:34 DEBUG ] 22:46:34 DEBUG "instance": "10.142.0.107:9100" 22:46:34 DEBUG } 22:46:34 DEBUG }, 22:46:34 DEBUG } 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG [...] 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG 22:46:34 DEBUG "instance": "10.142.0.43:9100" 22:46:34 DEBUG }, 22:46:34 DEBUG "value": [ 22:46:34 DEBUG 1713649594, 22:46:34 DEBUG "100509245.93333334" 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG ] 22:46:34 DEBUG } 22:46:34 DEBUG } 22:46:35 INFO 22:46:35 INFO [loop_until]: kubectl --namespace=xlou top pods 22:46:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:46:35 INFO [loop_until]: OK (rc = 0) 22:46:35 DEBUG --- stdout --- 22:46:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22233m 8777Mi am-767f6b67b5-qcqz2 21547m 8898Mi am-767f6b67b5-vbltz 24091m 8938Mi ds-cts-0 20144m 12276Mi ds-cts-1 19661m 12397Mi ds-cts-2 20903m 12536Mi ds-idrepo-0 9743m 3923Mi ds-idrepo-1 9805m 3955Mi ds-idrepo-2 9563m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 4m 1238Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11828m 3386Mi 22:46:35 DEBUG --- stderr --- 22:46:35 DEBUG 22:46:35 INFO 22:46:35 INFO [loop_until]: kubectl --namespace=xlou top node 22:46:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:46:35 INFO [loop_until]: OK (rc = 0) 22:46:35 DEBUG --- stdout --- 22:46:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6282Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9943m 33% 7678Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21876m 73% 11502Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3364Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20212m 67% 15508Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5210Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22382m 74% 11807Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19811m 66% 3135Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24826m 83% 11201Mi 10% gke-xlou-cdm-ds-0f601751-j488 10262m 34% 7291Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19983m 66% 17419Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10045m 33% 7930Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20424m 68% 15237Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11540m 72% 11272Mi 19% 22:46:35 DEBUG --- stderr --- 22:46:35 DEBUG 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG --- http response --- 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG { 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG "data": { 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG "result": [ 22:47:34 DEBUG { 22:47:34 DEBUG "metric": { 22:47:34 DEBUG "pod": "ds-idrepo-0" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG [...] 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "pod": "ds-cts-2" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG "0" 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649654" 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG --- http response --- 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649654" 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG { 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG "data": { 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG "result": [ 22:47:34 DEBUG { 22:47:34 DEBUG "metric": { 22:47:34 DEBUG "pod": "ds-idrepo-0" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG [...] 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "pod": "ds-idrepo-2" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG "0" 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649654" 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 ERROR [http_cmd]: ERROR 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG --- http response --- 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG --- http response --- 22:47:34 ERROR --- status code --- 22:47:34 DEBUG --- http response --- 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:47:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:34 DEBUG --- http response --- 22:47:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:47:34 DEBUG ----- output ----- 22:47:34 ERROR --- http response --- 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:34 DEBUG { 22:47:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:47:34 DEBUG { 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG "status": "success", 22:47:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG { 22:47:34 DEBUG "data": { 22:47:34 DEBUG "data": { 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG "data": { 22:47:34 DEBUG "result": [ 22:47:34 DEBUG "result": [ 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG { 22:47:34 DEBUG { 22:47:34 DEBUG "result": [ 22:47:34 DEBUG "metric": { 22:47:34 DEBUG "metric": { 22:47:34 DEBUG { 22:47:34 DEBUG "pod": "ds-cts-0" 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:47:34 DEBUG "metric": { 22:47:34 DEBUG }, 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG }, 22:47:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:47:34 DEBUG "value": [ 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG "value": [ 22:47:34 DEBUG }, 22:47:34 DEBUG 22:47:34 DEBUG --- http response --- 22:47:34 DEBUG 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG 22:47:34 DEBUG [...] 22:47:34 DEBUG [...] 22:47:34 DEBUG { 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG [...] 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "data": { 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG 22:47:34 DEBUG "pod": "ds-cts-2" 22:47:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:47:34 DEBUG "result": [ 22:47:34 DEBUG 22:47:34 DEBUG }, 22:47:34 DEBUG }, 22:47:34 DEBUG { 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:47:34 DEBUG "value": [ 22:47:34 DEBUG "value": [ 22:47:34 DEBUG "metric": { 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG }, 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG "pod": "ds-idrepo-0" 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG "value": [ 22:47:34 DEBUG "0.5431000000000002" 22:47:34 DEBUG "0" 22:47:34 DEBUG }, 22:47:34 DEBUG --- http response --- 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG ] 22:47:34 DEBUG ] 22:47:34 DEBUG "value": [ 22:47:34 DEBUG "3645894656" 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 DEBUG 22:47:34 DEBUG ] 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG ] 22:47:34 DEBUG ] 22:47:34 DEBUG 22:47:34 DEBUG } 22:47:34 DEBUG { 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 DEBUG 22:47:34 DEBUG ] 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 DEBUG [...] 22:47:34 DEBUG } 22:47:34 DEBUG "data": { 22:47:34 DEBUG 22:47:34 DEBUG } 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG 22:47:34 DEBUG "result": [ 22:47:34 DEBUG 22:47:34 DEBUG { 22:47:34 DEBUG "pod": "ds-idrepo-2" 22:47:34 DEBUG "metric": { 22:47:34 DEBUG }, 22:47:34 DEBUG "instance": "10.142.0.107:9100" 22:47:34 DEBUG "value": [ 22:47:34 DEBUG }, 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG "0" 22:47:34 DEBUG 22:47:34 DEBUG ] 22:47:34 DEBUG 22:47:34 DEBUG } 22:47:34 DEBUG 22:47:34 DEBUG ] 22:47:34 DEBUG [...] 22:47:34 DEBUG } 22:47:34 DEBUG 22:47:34 DEBUG } 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "instance": "10.142.0.43:9100" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG "0.09533333333334516" 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG --- http response --- 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG { 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG "data": { 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG "result": [ 22:47:34 DEBUG { 22:47:34 DEBUG "metric": { 22:47:34 DEBUG "pod": "ds-idrepo-0" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG [...] 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "pod": "ds-idrepo-2" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG "22546.71387778412" 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG --- http response --- 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG { 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG "data": { 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG "result": [ 22:47:34 DEBUG { 22:47:34 DEBUG "metric": { 22:47:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG [...] 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG "3342.4666666666667" 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG --- http response --- 22:47:34 DEBUG --- http response --- 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG { 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG "data": { 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG "result": [ 22:47:34 DEBUG { 22:47:34 DEBUG "metric": { 22:47:34 DEBUG "pod": "ds-cts-0" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG [...] 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "pod": "ds-cts-2" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG "0" 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG { 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG "data": { 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG "result": [ 22:47:34 DEBUG { 22:47:34 DEBUG "metric": { 22:47:34 DEBUG "pod": "ds-cts-0" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG [...] 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "pod": "ds-cts-2" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG "32.86666666666667" 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG --- http response --- 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG { 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG "data": { 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG "result": [ 22:47:34 DEBUG { 22:47:34 DEBUG "metric": { 22:47:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG [...] 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG "3336.2666666666664" 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG --- http response --- 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG { 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG "data": { 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG "result": [ 22:47:34 DEBUG { 22:47:34 DEBUG "metric": { 22:47:34 DEBUG "pod": "ds-idrepo-0" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG [...] 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "pod": "ds-idrepo-2" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG "0" 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649654" 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG --- http response --- 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG { 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG "data": { 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG "result": [ 22:47:34 DEBUG { 22:47:34 DEBUG "metric": { 22:47:34 DEBUG "pod": "ds-idrepo-0" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG [...] 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "pod": "ds-idrepo-2" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG "2437.482581382067" 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG --- http response --- 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG { 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG "data": { 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG "result": [ 22:47:34 DEBUG { 22:47:34 DEBUG "metric": { 22:47:34 DEBUG "instance": "10.142.0.107:9100" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG [...] 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "instance": "10.142.0.43:9100" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG "0.44010000000000393" 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: http status code OK 22:47:34 DEBUG --- status code --- 22:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:34 DEBUG --- http response --- 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649654" 22:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:34 DEBUG ----- output ----- 22:47:34 DEBUG { 22:47:34 DEBUG "status": "success", 22:47:34 DEBUG "data": { 22:47:34 DEBUG "resultType": "vector", 22:47:34 DEBUG "result": [ 22:47:34 DEBUG { 22:47:34 DEBUG "metric": { 22:47:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG [...] 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG 22:47:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:47:34 DEBUG }, 22:47:34 DEBUG "value": [ 22:47:34 DEBUG 1713649654, 22:47:34 DEBUG "3335.733333333333" 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG ] 22:47:34 DEBUG } 22:47:34 DEBUG } 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649654" 22:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649654" 22:47:35 INFO [http_cmd]: http status code OK 22:47:35 INFO [http_cmd]: http status code OK 22:47:35 DEBUG --- status code --- 22:47:35 DEBUG --- status code --- 22:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:35 DEBUG --- http response --- 22:47:35 DEBUG --- http response --- 22:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:35 DEBUG ----- output ----- 22:47:35 DEBUG ----- output ----- 22:47:35 DEBUG { 22:47:35 DEBUG { 22:47:35 DEBUG "status": "success", 22:47:35 DEBUG "status": "success", 22:47:35 DEBUG "data": { 22:47:35 DEBUG "data": { 22:47:35 DEBUG "resultType": "vector", 22:47:35 DEBUG "resultType": "vector", 22:47:35 DEBUG "result": [ 22:47:35 DEBUG "result": [ 22:47:35 DEBUG { 22:47:35 DEBUG { 22:47:35 DEBUG "metric": { 22:47:35 DEBUG "metric": { 22:47:35 DEBUG "instance": "10.142.0.107:9100" 22:47:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:47:35 DEBUG }, 22:47:35 DEBUG }, 22:47:35 DEBUG "value": [ 22:47:35 DEBUG "value": [ 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG [...] 22:47:35 DEBUG [...] 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:47:35 DEBUG "instance": "10.142.0.43:9100" 22:47:35 DEBUG }, 22:47:35 DEBUG }, 22:47:35 DEBUG "value": [ 22:47:35 DEBUG "value": [ 22:47:35 DEBUG 1713649654, 22:47:35 DEBUG 1713649654, 22:47:35 DEBUG "101.875" 22:47:35 DEBUG "100132160.36666667" 22:47:35 DEBUG ] 22:47:35 DEBUG ] 22:47:35 DEBUG } 22:47:35 DEBUG } 22:47:35 DEBUG ] 22:47:35 DEBUG ] 22:47:35 DEBUG } 22:47:35 DEBUG } 22:47:35 DEBUG } 22:47:35 DEBUG } 22:47:35 INFO [http_cmd]: http status code OK 22:47:35 DEBUG --- status code --- 22:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:35 DEBUG --- http response --- 22:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:35 DEBUG ----- output ----- 22:47:35 DEBUG { 22:47:35 DEBUG "status": "success", 22:47:35 DEBUG "data": { 22:47:35 DEBUG "resultType": "vector", 22:47:35 DEBUG "result": [ 22:47:35 DEBUG { 22:47:35 DEBUG "metric": { 22:47:35 DEBUG "pod": "ds-idrepo-0" 22:47:35 DEBUG }, 22:47:35 DEBUG "value": [ 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 INFO [http_cmd]: http status code OK 22:47:35 DEBUG 22:47:35 DEBUG --- status code --- 22:47:35 DEBUG [...] 22:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:35 DEBUG 22:47:35 DEBUG --- http response --- 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG "pod": "ds-idrepo-2" 22:47:35 DEBUG }, 22:47:35 DEBUG "value": [ 22:47:35 DEBUG 1713649654, 22:47:35 DEBUG "4.39385796316368" 22:47:35 DEBUG ] 22:47:35 DEBUG } 22:47:35 DEBUG ] 22:47:35 DEBUG } 22:47:35 DEBUG } 22:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:35 DEBUG ----- output ----- 22:47:35 DEBUG { 22:47:35 DEBUG "status": "success", 22:47:35 DEBUG "data": { 22:47:35 DEBUG "resultType": "vector", 22:47:35 DEBUG "result": [ 22:47:35 DEBUG { 22:47:35 DEBUG "metric": { 22:47:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:47:35 DEBUG }, 22:47:35 DEBUG "value": [ 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG [...] 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:47:35 DEBUG }, 22:47:35 DEBUG "value": [ 22:47:35 DEBUG 1713649654, 22:47:35 DEBUG "11.666452944830592" 22:47:35 DEBUG ] 22:47:35 DEBUG } 22:47:35 DEBUG ] 22:47:35 DEBUG } 22:47:35 DEBUG } 22:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649655" 22:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649655" 22:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649655" 22:47:35 INFO [http_cmd]: http status code OK 22:47:35 DEBUG --- status code --- 22:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:35 DEBUG --- http response --- 22:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649655" 22:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649655" 22:47:35 DEBUG ----- output ----- 22:47:35 DEBUG { 22:47:35 DEBUG "status": "success", 22:47:35 DEBUG "data": { 22:47:35 DEBUG "resultType": "vector", 22:47:35 DEBUG "result": [ 22:47:35 DEBUG { 22:47:35 DEBUG "metric": { 22:47:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:47:35 DEBUG }, 22:47:35 DEBUG "value": [ 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG [...] 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:47:35 DEBUG }, 22:47:35 DEBUG "value": [ 22:47:35 DEBUG 1713649655, 22:47:35 DEBUG "3333.0666666666666" 22:47:35 DEBUG ] 22:47:35 DEBUG } 22:47:35 DEBUG ] 22:47:35 DEBUG } 22:47:35 DEBUG } 22:47:35 INFO [http_cmd]: http status code OK 22:47:35 DEBUG --- status code --- 22:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:35 DEBUG --- http response --- 22:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:35 DEBUG ----- output ----- 22:47:35 DEBUG { 22:47:35 DEBUG "status": "success", 22:47:35 DEBUG "data": { 22:47:35 DEBUG "resultType": "vector", 22:47:35 DEBUG "result": [ 22:47:35 DEBUG { 22:47:35 DEBUG "metric": { 22:47:35 DEBUG "instance": "10.142.0.107:9100" 22:47:35 DEBUG }, 22:47:35 DEBUG "value": [ 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG [...] 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG "instance": "10.142.0.43:9100" 22:47:35 DEBUG }, 22:47:35 DEBUG "value": [ 22:47:35 DEBUG 1713649655, 22:47:35 DEBUG "4193751374848" 22:47:35 DEBUG ] 22:47:35 DEBUG } 22:47:35 DEBUG ] 22:47:35 DEBUG } 22:47:35 DEBUG } 22:47:35 INFO [http_cmd]: http status code OK 22:47:35 INFO [http_cmd]: http status code OK 22:47:35 DEBUG --- status code --- 22:47:35 DEBUG --- status code --- 22:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:35 DEBUG --- http response --- 22:47:35 DEBUG --- http response --- 22:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:47:35 DEBUG ----- output ----- 22:47:35 DEBUG ----- output ----- 22:47:35 DEBUG { 22:47:35 DEBUG { 22:47:35 DEBUG "status": "success", 22:47:35 DEBUG "status": "success", 22:47:35 DEBUG "data": { 22:47:35 DEBUG "data": { 22:47:35 DEBUG "resultType": "vector", 22:47:35 DEBUG "resultType": "vector", 22:47:35 DEBUG "result": [ 22:47:35 DEBUG "result": [ 22:47:35 DEBUG { 22:47:35 DEBUG { 22:47:35 DEBUG "metric": { 22:47:35 DEBUG "metric": { 22:47:35 DEBUG "pod": "ds-idrepo-0" 22:47:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:47:35 DEBUG }, 22:47:35 DEBUG }, 22:47:35 DEBUG "value": [ 22:47:35 DEBUG "value": [ 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG [...] 22:47:35 DEBUG [...] 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG "pod": "ds-idrepo-2" 22:47:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:47:35 DEBUG }, 22:47:35 DEBUG }, 22:47:35 DEBUG "value": [ 22:47:35 INFO [http_cmd]: http status code OK 22:47:35 DEBUG "value": [ 22:47:35 DEBUG 1713649655, 22:47:35 DEBUG --- status code --- 22:47:35 DEBUG 1713649655, 22:47:35 DEBUG "0" 22:47:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:47:35 DEBUG "600360.5" 22:47:35 DEBUG ] 22:47:35 DEBUG --- http response --- 22:47:35 DEBUG ] 22:47:35 DEBUG } 22:47:35 DEBUG } 22:47:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:47:35 DEBUG ] 22:47:35 DEBUG ] 22:47:35 DEBUG ----- output ----- 22:47:35 DEBUG } 22:47:35 DEBUG } 22:47:35 DEBUG { 22:47:35 DEBUG } 22:47:35 DEBUG } 22:47:35 DEBUG "status": "success", 22:47:35 DEBUG "data": { 22:47:35 DEBUG "resultType": "vector", 22:47:35 DEBUG "result": [ 22:47:35 DEBUG { 22:47:35 DEBUG "metric": { 22:47:35 DEBUG "instance": "10.142.0.107:9100" 22:47:35 DEBUG }, 22:47:35 DEBUG "value": [ 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG [...] 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG 22:47:35 DEBUG "instance": "10.142.0.43:9100" 22:47:35 DEBUG }, 22:47:35 DEBUG "value": [ 22:47:35 DEBUG 1713649655, 22:47:35 DEBUG "100133146.4" 22:47:35 DEBUG ] 22:47:35 DEBUG } 22:47:35 DEBUG ] 22:47:35 DEBUG } 22:47:35 DEBUG } 22:47:35 INFO 22:47:35 INFO [loop_until]: kubectl --namespace=xlou top pods 22:47:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:47:35 INFO [loop_until]: OK (rc = 0) 22:47:35 DEBUG --- stdout --- 22:47:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22577m 8776Mi am-767f6b67b5-qcqz2 21870m 8898Mi am-767f6b67b5-vbltz 24164m 8937Mi ds-cts-0 20036m 12296Mi ds-cts-1 19907m 12399Mi ds-cts-2 20490m 12544Mi ds-idrepo-0 9472m 3923Mi ds-idrepo-1 9615m 3956Mi ds-idrepo-2 9774m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-5497cb87d7-mlld4 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11567m 3385Mi 22:47:35 DEBUG --- stderr --- 22:47:35 DEBUG 22:47:35 INFO 22:47:35 INFO [loop_until]: kubectl --namespace=xlou top node 22:47:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:47:35 INFO [loop_until]: OK (rc = 0) 22:47:35 DEBUG --- stdout --- 22:47:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6294Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9966m 33% 7691Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21984m 73% 11510Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3361Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20207m 67% 15533Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5232Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22371m 74% 11805Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19779m 66% 3124Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24665m 82% 11188Mi 10% gke-xlou-cdm-ds-0f601751-j488 10255m 34% 7284Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19713m 66% 17419Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10028m 33% 7927Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20388m 68% 15253Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11944m 75% 11276Mi 19% 22:47:35 DEBUG --- stderr --- 22:47:35 DEBUG 22:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649714" 22:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649714" 22:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649714" 22:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649714" 22:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649714" 22:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649714" 22:48:34 INFO [http_cmd]: http status code OK 22:48:34 DEBUG --- status code --- 22:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:34 DEBUG --- http response --- 22:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649714" 22:48:34 INFO [http_cmd]: http status code OK 22:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:34 DEBUG --- status code --- 22:48:34 INFO [http_cmd]: http status code OK 22:48:34 DEBUG ----- output ----- 22:48:34 INFO [http_cmd]: http status code OK 22:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:34 DEBUG --- status code --- 22:48:34 DEBUG { 22:48:34 DEBUG --- status code --- 22:48:34 DEBUG --- http response --- 22:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:34 DEBUG "status": "success", 22:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:34 DEBUG --- http response --- 22:48:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:48:34 DEBUG "data": { 22:48:34 DEBUG --- http response --- 22:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:34 DEBUG "resultType": "vector", 22:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:34 DEBUG ----- output ----- 22:48:34 DEBUG "result": [ 22:48:34 DEBUG ----- output ----- 22:48:34 DEBUG { 22:48:34 DEBUG { 22:48:34 DEBUG { 22:48:34 DEBUG "status": "success", 22:48:34 DEBUG "metric": { 22:48:34 DEBUG "status": "success", 22:48:34 DEBUG "data": { 22:48:34 DEBUG "pod": "ds-idrepo-0" 22:48:34 DEBUG "data": { 22:48:34 DEBUG "resultType": "vector", 22:48:34 DEBUG }, 22:48:34 DEBUG "resultType": "vector", 22:48:34 DEBUG "result": [ 22:48:34 DEBUG "value": [ 22:48:34 DEBUG "result": [ 22:48:34 DEBUG { 22:48:34 DEBUG 22:48:34 DEBUG { 22:48:34 DEBUG "metric": { 22:48:34 DEBUG 22:48:34 DEBUG "metric": { 22:48:34 DEBUG "instance": "10.142.0.107:9100" 22:48:34 DEBUG 22:48:34 DEBUG "pod": "ds-idrepo-1" 22:48:34 DEBUG }, 22:48:34 DEBUG [...] 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG "pod": "ds-idrepo-2" 22:48:34 DEBUG 22:48:34 DEBUG [...] 22:48:34 DEBUG }, 22:48:34 DEBUG [...] 22:48:34 DEBUG 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 1713649714, 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG "0" 22:48:34 DEBUG 22:48:34 DEBUG "instance": "10.142.0.43:9100" 22:48:34 DEBUG ] 22:48:34 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:48:34 DEBUG }, 22:48:34 DEBUG } 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG ] 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 1713649714, 22:48:34 DEBUG } 22:48:34 DEBUG 1713649714, 22:48:34 DEBUG "0.0870000000000156" 22:48:34 DEBUG } 22:48:34 DEBUG "0" 22:48:34 DEBUG ] 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG } 22:48:34 DEBUG ] 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG } 22:48:34 DEBUG } 22:48:34 DEBUG } 22:48:34 ERROR [http_cmd]: ERROR 22:48:34 INFO [http_cmd]: http status code OK 22:48:34 ERROR --- status code --- 22:48:34 DEBUG --- status code --- 22:48:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:34 ERROR --- http response --- 22:48:34 DEBUG --- http response --- 22:48:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:48:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:34 DEBUG ----- output ----- 22:48:34 DEBUG { 22:48:34 DEBUG "status": "success", 22:48:34 DEBUG "data": { 22:48:34 DEBUG "resultType": "vector", 22:48:34 DEBUG "result": [ 22:48:34 DEBUG { 22:48:34 DEBUG "metric": { 22:48:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 INFO [http_cmd]: http status code OK 22:48:34 DEBUG 22:48:34 DEBUG --- status code --- 22:48:34 DEBUG [...] 22:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:34 DEBUG 22:48:34 DEBUG --- http response --- 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 1713649714, 22:48:34 DEBUG "0" 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:48:34 DEBUG ] 22:48:34 DEBUG ----- output ----- 22:48:34 DEBUG } 22:48:34 DEBUG { 22:48:34 DEBUG } 22:48:34 DEBUG "status": "success", 22:48:34 DEBUG "data": { 22:48:34 DEBUG "resultType": "vector", 22:48:34 DEBUG "result": [ 22:48:34 DEBUG { 22:48:34 DEBUG "metric": { 22:48:34 DEBUG "pod": "ds-idrepo-0" 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG [...] 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG "pod": "ds-cts-2" 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 1713649714, 22:48:34 DEBUG "0" 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG } 22:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649714" 22:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649714" 22:48:34 INFO [http_cmd]: http status code OK 22:48:34 DEBUG --- status code --- 22:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:34 DEBUG --- http response --- 22:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:48:34 DEBUG ----- output ----- 22:48:34 DEBUG { 22:48:34 DEBUG "status": "success", 22:48:34 DEBUG "data": { 22:48:34 DEBUG "resultType": "vector", 22:48:34 DEBUG "result": [ 22:48:34 DEBUG { 22:48:34 DEBUG "metric": { 22:48:34 DEBUG "pod": "ds-cts-0" 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG [...] 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG "pod": "ds-cts-2" 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 1713649714, 22:48:34 DEBUG "0.4957333333333338" 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG } 22:48:34 INFO [http_cmd]: http status code OK 22:48:34 DEBUG --- status code --- 22:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:34 DEBUG --- http response --- 22:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:34 DEBUG ----- output ----- 22:48:34 DEBUG { 22:48:34 DEBUG "status": "success", 22:48:34 DEBUG "data": { 22:48:34 DEBUG "resultType": "vector", 22:48:34 DEBUG "result": [ 22:48:34 DEBUG { 22:48:34 DEBUG "metric": { 22:48:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG [...] 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 1713649714, 22:48:34 DEBUG "3649433600" 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG } 22:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649714" 22:48:34 INFO [http_cmd]: http status code OK 22:48:34 DEBUG --- status code --- 22:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:34 DEBUG --- http response --- 22:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:34 DEBUG ----- output ----- 22:48:34 DEBUG { 22:48:34 DEBUG "status": "success", 22:48:34 DEBUG "data": { 22:48:34 DEBUG "resultType": "vector", 22:48:34 DEBUG "result": [ 22:48:34 DEBUG { 22:48:34 DEBUG "metric": { 22:48:34 DEBUG "pod": "ds-idrepo-1" 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG [...] 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 1713649714, 22:48:34 DEBUG "0" 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG } 22:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649714" 22:48:34 INFO [http_cmd]: http status code OK 22:48:34 DEBUG --- status code --- 22:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:34 DEBUG --- http response --- 22:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:34 DEBUG ----- output ----- 22:48:34 DEBUG { 22:48:34 DEBUG "status": "success", 22:48:34 DEBUG "data": { 22:48:34 DEBUG "resultType": "vector", 22:48:34 DEBUG "result": [ 22:48:34 DEBUG { 22:48:34 DEBUG "metric": { 22:48:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG [...] 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 1713649714, 22:48:34 DEBUG "3338.366666666667" 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG } 22:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649714" 22:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649714" 22:48:34 INFO [http_cmd]: http status code OK 22:48:34 INFO [http_cmd]: http status code OK 22:48:34 DEBUG --- status code --- 22:48:34 DEBUG --- status code --- 22:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:34 DEBUG --- http response --- 22:48:34 DEBUG --- http response --- 22:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:48:34 DEBUG ----- output ----- 22:48:34 DEBUG { 22:48:34 DEBUG "status": "success", 22:48:34 DEBUG "data": { 22:48:34 DEBUG "resultType": "vector", 22:48:34 DEBUG "result": [ 22:48:34 DEBUG { 22:48:34 DEBUG "metric": { 22:48:34 DEBUG "pod": "ds-cts-0" 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG [...] 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:48:34 DEBUG 22:48:34 DEBUG ----- output ----- 22:48:34 DEBUG "pod": "ds-cts-2" 22:48:34 DEBUG { 22:48:34 DEBUG }, 22:48:34 DEBUG "status": "success", 22:48:34 DEBUG "value": [ 22:48:34 DEBUG "data": { 22:48:34 DEBUG 1713649714, 22:48:34 DEBUG "resultType": "vector", 22:48:34 DEBUG "0" 22:48:34 DEBUG "result": [ 22:48:34 DEBUG ] 22:48:34 DEBUG { 22:48:34 DEBUG } 22:48:34 DEBUG "metric": { 22:48:34 DEBUG ] 22:48:34 DEBUG "pod": "ds-cts-0" 22:48:34 DEBUG } 22:48:34 DEBUG }, 22:48:34 DEBUG } 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG [...] 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG 22:48:34 DEBUG "pod": "ds-cts-2" 22:48:34 DEBUG }, 22:48:34 DEBUG "value": [ 22:48:34 DEBUG 1713649714, 22:48:34 DEBUG "28.733333333333334" 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG ] 22:48:34 DEBUG } 22:48:34 DEBUG } 22:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649715" 22:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649715" 22:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649715" 22:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649715" 22:48:35 INFO [http_cmd]: http status code OK 22:48:35 DEBUG --- status code --- 22:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:35 DEBUG --- http response --- 22:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:35 DEBUG ----- output ----- 22:48:35 DEBUG { 22:48:35 DEBUG "status": "success", 22:48:35 DEBUG "data": { 22:48:35 DEBUG "resultType": "vector", 22:48:35 DEBUG "result": [ 22:48:35 DEBUG { 22:48:35 DEBUG "metric": { 22:48:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG [...] 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 1713649715, 22:48:35 DEBUG "3335.5666666666666" 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG } 22:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649715" 22:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649715" 22:48:35 INFO [http_cmd]: http status code OK 22:48:35 INFO [http_cmd]: http status code OK 22:48:35 DEBUG --- status code --- 22:48:35 DEBUG --- status code --- 22:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:35 DEBUG --- http response --- 22:48:35 DEBUG --- http response --- 22:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649715" 22:48:35 DEBUG ----- output ----- 22:48:35 INFO [http_cmd]: http status code OK 22:48:35 DEBUG ----- output ----- 22:48:35 DEBUG { 22:48:35 DEBUG --- status code --- 22:48:35 DEBUG { 22:48:35 DEBUG "status": "success", 22:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:35 DEBUG "status": "success", 22:48:35 DEBUG "data": { 22:48:35 DEBUG --- http response --- 22:48:35 DEBUG "data": { 22:48:35 DEBUG "resultType": "vector", 22:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:35 DEBUG "resultType": "vector", 22:48:35 DEBUG "result": [ 22:48:35 DEBUG ----- output ----- 22:48:35 DEBUG "result": [ 22:48:35 DEBUG { 22:48:35 DEBUG { 22:48:35 DEBUG { 22:48:35 DEBUG "metric": { 22:48:35 DEBUG "status": "success", 22:48:35 DEBUG "metric": { 22:48:35 DEBUG "pod": "ds-idrepo-1" 22:48:35 DEBUG "data": { 22:48:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:48:35 DEBUG }, 22:48:35 DEBUG "resultType": "vector", 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG "result": [ 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 22:48:35 DEBUG { 22:48:35 INFO [http_cmd]: http status code OK 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG "metric": { 22:48:35 DEBUG --- status code --- 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG "pod": "ds-idrepo-0" 22:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:35 DEBUG 22:48:35 DEBUG [...] 22:48:35 DEBUG }, 22:48:35 DEBUG --- http response --- 22:48:35 DEBUG [...] 22:48:35 DEBUG 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 22:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 INFO [http_cmd]: http status code OK 22:48:35 DEBUG 22:48:35 DEBUG ----- output ----- 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG --- status code --- 22:48:35 DEBUG 22:48:35 DEBUG { 22:48:35 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:48:35 DEBUG 22:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:48:35 DEBUG "status": "success", 22:48:35 DEBUG }, 22:48:35 DEBUG [...] 22:48:35 DEBUG --- http response --- 22:48:35 DEBUG }, 22:48:35 DEBUG "data": { 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 22:48:35 DEBUG "value": [ 22:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:35 DEBUG "resultType": "vector", 22:48:35 DEBUG 1713649715, 22:48:35 DEBUG 22:48:35 DEBUG 1713649715, 22:48:35 DEBUG ----- output ----- 22:48:35 DEBUG "result": [ 22:48:35 DEBUG "0.000020349141085052367" 22:48:35 DEBUG 22:48:35 DEBUG "11.713650929540561" 22:48:35 DEBUG { 22:48:35 DEBUG { 22:48:35 DEBUG ] 22:48:35 DEBUG "pod": "ds-idrepo-2" 22:48:35 DEBUG ] 22:48:35 DEBUG "status": "success", 22:48:35 DEBUG "metric": { 22:48:35 DEBUG } 22:48:35 DEBUG }, 22:48:35 DEBUG } 22:48:35 DEBUG "data": { 22:48:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:48:35 DEBUG ] 22:48:35 DEBUG "value": [ 22:48:35 DEBUG ] 22:48:35 DEBUG "resultType": "vector", 22:48:35 DEBUG }, 22:48:35 DEBUG } 22:48:35 DEBUG 1713649715, 22:48:35 DEBUG } 22:48:35 DEBUG "result": [ 22:48:35 DEBUG "value": [ 22:48:35 DEBUG } 22:48:35 DEBUG "0" 22:48:35 DEBUG } 22:48:35 DEBUG { 22:48:35 DEBUG 22:48:35 DEBUG ] 22:48:35 DEBUG "metric": { 22:48:35 DEBUG 22:48:35 DEBUG } 22:48:35 DEBUG "pod": "ds-idrepo-1" 22:48:35 DEBUG 22:48:35 DEBUG ] 22:48:35 DEBUG }, 22:48:35 DEBUG [...] 22:48:35 DEBUG } 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 22:48:35 DEBUG } 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:48:35 INFO [http_cmd]: http status code OK 22:48:35 DEBUG [...] 22:48:35 DEBUG }, 22:48:35 DEBUG --- status code --- 22:48:35 DEBUG 22:48:35 DEBUG "value": [ 22:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:35 DEBUG 22:48:35 DEBUG 1713649715, 22:48:35 DEBUG --- http response --- 22:48:35 DEBUG 22:48:35 DEBUG "3335.366666666667" 22:48:35 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:35 DEBUG ] 22:48:35 DEBUG }, 22:48:35 DEBUG ----- output ----- 22:48:35 DEBUG } 22:48:35 DEBUG "value": [ 22:48:35 DEBUG { 22:48:35 DEBUG ] 22:48:35 DEBUG 1713649715, 22:48:35 DEBUG "status": "success", 22:48:35 DEBUG } 22:48:35 DEBUG "0" 22:48:35 DEBUG "data": { 22:48:35 DEBUG } 22:48:35 DEBUG ] 22:48:35 DEBUG "resultType": "vector", 22:48:35 DEBUG } 22:48:35 DEBUG "result": [ 22:48:35 DEBUG ] 22:48:35 DEBUG { 22:48:35 DEBUG } 22:48:35 DEBUG "metric": { 22:48:35 DEBUG } 22:48:35 DEBUG "instance": "10.142.0.107:9100" 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG [...] 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG "instance": "10.142.0.43:9100" 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 1713649715, 22:48:35 DEBUG "0.40390000000007265" 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG } 22:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649715" 22:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649715" 22:48:35 INFO [http_cmd]: http status code OK 22:48:35 INFO [http_cmd]: http status code OK 22:48:35 DEBUG --- status code --- 22:48:35 DEBUG --- status code --- 22:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:35 DEBUG --- http response --- 22:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649715" 22:48:35 DEBUG --- http response --- 22:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:35 DEBUG ----- output ----- 22:48:35 DEBUG ----- output ----- 22:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649715" 22:48:35 DEBUG { 22:48:35 DEBUG { 22:48:35 DEBUG "status": "success", 22:48:35 DEBUG "status": "success", 22:48:35 DEBUG "data": { 22:48:35 DEBUG "data": { 22:48:35 DEBUG "resultType": "vector", 22:48:35 DEBUG "resultType": "vector", 22:48:35 DEBUG "result": [ 22:48:35 DEBUG "result": [ 22:48:35 DEBUG { 22:48:35 DEBUG { 22:48:35 DEBUG "metric": { 22:48:35 DEBUG "metric": { 22:48:35 DEBUG "pod": "ds-idrepo-0" 22:48:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:48:35 DEBUG }, 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG [...] 22:48:35 DEBUG [...] 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG "pod": "ds-idrepo-2" 22:48:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:48:35 DEBUG }, 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 1713649715, 22:48:35 DEBUG 1713649715, 22:48:35 DEBUG "0" 22:48:35 DEBUG "600464.5" 22:48:35 DEBUG ] 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG } 22:48:35 DEBUG ] 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG } 22:48:35 DEBUG } 22:48:35 DEBUG } 22:48:35 INFO [http_cmd]: http status code OK 22:48:35 DEBUG --- status code --- 22:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:35 DEBUG --- http response --- 22:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:35 DEBUG ----- output ----- 22:48:35 DEBUG { 22:48:35 DEBUG "status": "success", 22:48:35 DEBUG "data": { 22:48:35 DEBUG "resultType": "vector", 22:48:35 DEBUG "result": [ 22:48:35 DEBUG { 22:48:35 DEBUG "metric": { 22:48:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG [...] 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 1713649715, 22:48:35 DEBUG "101.875" 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG } 22:48:35 INFO [http_cmd]: http status code OK 22:48:35 DEBUG --- status code --- 22:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:35 DEBUG --- http response --- 22:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:35 DEBUG ----- output ----- 22:48:35 DEBUG { 22:48:35 DEBUG "status": "success", 22:48:35 DEBUG "data": { 22:48:35 DEBUG "resultType": "vector", 22:48:35 DEBUG "result": [ 22:48:35 DEBUG { 22:48:35 DEBUG "metric": { 22:48:35 DEBUG "instance": "10.142.0.107:9100" 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG [...] 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG "instance": "10.142.0.43:9100" 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 1713649715, 22:48:35 DEBUG "100319050.93333334" 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG } 22:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649715" 22:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649715" 22:48:35 INFO [http_cmd]: http status code OK 22:48:35 DEBUG --- status code --- 22:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:35 DEBUG --- http response --- 22:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:48:35 DEBUG ----- output ----- 22:48:35 DEBUG { 22:48:35 DEBUG "status": "success", 22:48:35 DEBUG "data": { 22:48:35 DEBUG "resultType": "vector", 22:48:35 DEBUG "result": [ 22:48:35 DEBUG { 22:48:35 DEBUG "metric": { 22:48:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG [...] 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 1713649715, 22:48:35 DEBUG "3334.9333333333334" 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG } 22:48:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649715" 22:48:35 INFO [http_cmd]: http status code OK 22:48:35 DEBUG --- status code --- 22:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:35 DEBUG --- http response --- 22:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:35 DEBUG ----- output ----- 22:48:35 DEBUG { 22:48:35 DEBUG "status": "success", 22:48:35 DEBUG "data": { 22:48:35 DEBUG "resultType": "vector", 22:48:35 DEBUG "result": [ 22:48:35 DEBUG { 22:48:35 DEBUG "metric": { 22:48:35 DEBUG "instance": "10.142.0.107:9100" 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG [...] 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG "instance": "10.142.0.43:9100" 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 1713649715, 22:48:35 DEBUG "4199233842176" 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG } 22:48:35 INFO [http_cmd]: http status code OK 22:48:35 DEBUG --- status code --- 22:48:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:48:35 DEBUG --- http response --- 22:48:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:48:35 DEBUG ----- output ----- 22:48:35 DEBUG { 22:48:35 DEBUG "status": "success", 22:48:35 DEBUG "data": { 22:48:35 DEBUG "resultType": "vector", 22:48:35 DEBUG "result": [ 22:48:35 DEBUG { 22:48:35 DEBUG "metric": { 22:48:35 DEBUG "instance": "10.142.0.107:9100" 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG [...] 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG 22:48:35 DEBUG "instance": "10.142.0.43:9100" 22:48:35 DEBUG }, 22:48:35 DEBUG "value": [ 22:48:35 DEBUG 1713649715, 22:48:35 DEBUG "100320487.36666667" 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG ] 22:48:35 DEBUG } 22:48:35 DEBUG } 22:48:35 INFO 22:48:35 INFO [loop_until]: kubectl --namespace=xlou top pods 22:48:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:48:35 INFO [loop_until]: OK (rc = 0) 22:48:35 DEBUG --- stdout --- 22:48:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21918m 8777Mi am-767f6b67b5-qcqz2 22491m 8900Mi am-767f6b67b5-vbltz 24437m 8935Mi ds-cts-0 20343m 12295Mi ds-cts-1 19597m 12398Mi ds-cts-2 20667m 12548Mi ds-idrepo-0 9503m 3923Mi ds-idrepo-1 9658m 3955Mi ds-idrepo-2 9579m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1179Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-5497cb87d7-mlld4 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11594m 3390Mi 22:48:35 DEBUG --- stderr --- 22:48:35 DEBUG 22:48:35 INFO 22:48:35 INFO [loop_until]: kubectl --namespace=xlou top node 22:48:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:48:35 INFO [loop_until]: OK (rc = 0) 22:48:35 DEBUG --- stdout --- 22:48:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6286Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9988m 33% 7691Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21981m 73% 11520Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3373Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19418m 65% 15515Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5204Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22334m 74% 11801Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19743m 66% 3126Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24732m 82% 11188Mi 10% gke-xlou-cdm-ds-0f601751-j488 10291m 34% 7284Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19619m 65% 17411Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10029m 33% 7941Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20576m 68% 15256Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11957m 75% 11276Mi 19% 22:48:35 DEBUG --- stderr --- 22:48:35 DEBUG 22:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649774" 22:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649774" 22:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649774" 22:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649774" 22:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649774" 22:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649774" 22:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649774" 22:49:34 INFO [http_cmd]: http status code OK 22:49:34 DEBUG --- status code --- 22:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:34 ERROR [http_cmd]: ERROR 22:49:34 DEBUG --- http response --- 22:49:34 ERROR --- status code --- 22:49:34 INFO [http_cmd]: http status code OK 22:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649774" 22:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:49:34 DEBUG --- status code --- 22:49:34 DEBUG ----- output ----- 22:49:34 INFO [http_cmd]: http status code OK 22:49:34 INFO [http_cmd]: http status code OK 22:49:34 ERROR --- http response --- 22:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:34 DEBUG { 22:49:34 DEBUG --- status code --- 22:49:34 DEBUG --- status code --- 22:49:34 DEBUG --- http response --- 22:49:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:49:34 DEBUG "status": "success", 22:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:49:34 DEBUG "data": { 22:49:34 DEBUG --- http response --- 22:49:34 DEBUG --- http response --- 22:49:34 DEBUG ----- output ----- 22:49:34 DEBUG "resultType": "vector", 22:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:49:34 DEBUG { 22:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:34 DEBUG "result": [ 22:49:34 DEBUG ----- output ----- 22:49:34 DEBUG "status": "success", 22:49:34 DEBUG ----- output ----- 22:49:34 DEBUG { 22:49:34 DEBUG { 22:49:34 DEBUG "data": { 22:49:34 DEBUG { 22:49:34 DEBUG "metric": { 22:49:34 DEBUG "status": "success", 22:49:34 DEBUG "resultType": "vector", 22:49:34 DEBUG "status": "success", 22:49:34 DEBUG "pod": "ds-idrepo-0" 22:49:34 DEBUG "data": { 22:49:34 DEBUG "result": [ 22:49:34 DEBUG "data": { 22:49:34 DEBUG }, 22:49:34 DEBUG "resultType": "vector", 22:49:34 DEBUG { 22:49:34 DEBUG "resultType": "vector", 22:49:34 DEBUG "value": [ 22:49:34 DEBUG "result": [ 22:49:34 DEBUG "metric": { 22:49:34 DEBUG "result": [ 22:49:34 DEBUG 22:49:34 DEBUG { 22:49:34 DEBUG "pod": "ds-idrepo-0" 22:49:34 DEBUG { 22:49:34 DEBUG 22:49:34 DEBUG "metric": { 22:49:34 DEBUG }, 22:49:34 DEBUG "metric": { 22:49:34 DEBUG 22:49:34 DEBUG "pod": "ds-idrepo-1" 22:49:34 DEBUG "value": [ 22:49:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:49:34 DEBUG [...] 22:49:34 DEBUG }, 22:49:34 DEBUG 22:49:34 DEBUG }, 22:49:34 DEBUG 22:49:34 DEBUG "value": [ 22:49:34 DEBUG 22:49:34 DEBUG "value": [ 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG [...] 22:49:34 DEBUG 22:49:34 DEBUG "pod": "ds-idrepo-2" 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG }, 22:49:34 DEBUG [...] 22:49:34 DEBUG 22:49:34 DEBUG [...] 22:49:34 DEBUG "value": [ 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 1713649774, 22:49:34 DEBUG 22:49:34 DEBUG "pod": "ds-cts-2" 22:49:34 DEBUG 22:49:34 DEBUG "0" 22:49:34 DEBUG 22:49:34 DEBUG }, 22:49:34 DEBUG 22:49:34 DEBUG ] 22:49:34 DEBUG "pod": "lodemon-5497cb87d7-mlld4" 22:49:34 DEBUG "value": [ 22:49:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:49:34 DEBUG } 22:49:34 DEBUG }, 22:49:34 DEBUG 1713649774, 22:49:34 DEBUG }, 22:49:34 DEBUG ] 22:49:34 DEBUG "value": [ 22:49:34 DEBUG "0" 22:49:34 DEBUG "value": [ 22:49:34 DEBUG } 22:49:34 DEBUG 1713649774, 22:49:34 DEBUG ] 22:49:34 DEBUG 1713649774, 22:49:34 DEBUG } 22:49:34 DEBUG "0" 22:49:34 DEBUG } 22:49:34 DEBUG "0" 22:49:34 DEBUG ] 22:49:34 DEBUG ] 22:49:34 DEBUG ] 22:49:34 DEBUG } 22:49:34 DEBUG } 22:49:34 DEBUG } 22:49:34 DEBUG ] 22:49:34 DEBUG } 22:49:34 DEBUG ] 22:49:34 DEBUG } 22:49:34 DEBUG } 22:49:34 DEBUG } 22:49:34 DEBUG } 22:49:34 INFO [http_cmd]: http status code OK 22:49:34 DEBUG --- status code --- 22:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:34 DEBUG --- http response --- 22:49:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:49:34 INFO [http_cmd]: http status code OK 22:49:34 DEBUG --- status code --- 22:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:34 DEBUG --- http response --- 22:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:34 DEBUG ----- output ----- 22:49:34 DEBUG { 22:49:34 DEBUG "status": "success", 22:49:34 DEBUG "data": { 22:49:34 DEBUG "resultType": "vector", 22:49:34 DEBUG "result": [ 22:49:34 DEBUG { 22:49:34 DEBUG "metric": { 22:49:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:49:34 DEBUG }, 22:49:34 DEBUG "value": [ 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG [...] 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:49:34 DEBUG }, 22:49:34 DEBUG "value": [ 22:49:34 DEBUG 1713649774, 22:49:34 DEBUG "3655434240" 22:49:34 DEBUG ] 22:49:34 DEBUG } 22:49:34 DEBUG ] 22:49:34 DEBUG } 22:49:34 DEBUG } 22:49:34 INFO [http_cmd]: http status code OK 22:49:34 DEBUG --- status code --- 22:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:34 DEBUG --- http response --- 22:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:34 DEBUG ----- output ----- 22:49:34 DEBUG { 22:49:34 DEBUG "status": "success", 22:49:34 DEBUG "data": { 22:49:34 DEBUG "resultType": "vector", 22:49:34 DEBUG "result": [ 22:49:34 DEBUG { 22:49:34 DEBUG "metric": { 22:49:34 DEBUG "instance": "10.142.0.107:9100" 22:49:34 DEBUG }, 22:49:34 DEBUG "value": [ 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG [...] 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG "instance": "10.142.0.43:9100" 22:49:34 DEBUG }, 22:49:34 DEBUG "value": [ 22:49:34 DEBUG 1713649774, 22:49:34 DEBUG "0.08733333333333157" 22:49:34 DEBUG ] 22:49:34 DEBUG } 22:49:34 DEBUG ] 22:49:34 DEBUG } 22:49:34 DEBUG } 22:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649774" 22:49:34 INFO [http_cmd]: http status code OK 22:49:34 DEBUG --- status code --- 22:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:34 DEBUG --- http response --- 22:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:49:34 DEBUG ----- output ----- 22:49:34 DEBUG { 22:49:34 DEBUG "status": "success", 22:49:34 DEBUG "data": { 22:49:34 DEBUG "resultType": "vector", 22:49:34 DEBUG "result": [ 22:49:34 DEBUG { 22:49:34 DEBUG "metric": { 22:49:34 DEBUG "pod": "ds-cts-0" 22:49:34 DEBUG }, 22:49:34 DEBUG "value": [ 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG [...] 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG "pod": "ds-cts-2" 22:49:34 DEBUG }, 22:49:34 DEBUG "value": [ 22:49:34 DEBUG 1713649774, 22:49:34 DEBUG "0.5083666666666669" 22:49:34 DEBUG ] 22:49:34 DEBUG } 22:49:34 DEBUG ] 22:49:34 DEBUG } 22:49:34 DEBUG } 22:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649774" 22:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649774" 22:49:34 INFO [http_cmd]: http status code OK 22:49:34 DEBUG --- status code --- 22:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:34 DEBUG --- http response --- 22:49:34 INFO [http_cmd]: http status code OK 22:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:34 DEBUG --- status code --- 22:49:34 DEBUG ----- output ----- 22:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:34 DEBUG { 22:49:34 DEBUG --- http response --- 22:49:34 DEBUG "status": "success", 22:49:34 DEBUG "data": { 22:49:34 DEBUG "resultType": "vector", 22:49:34 DEBUG "result": [ 22:49:34 DEBUG { 22:49:34 DEBUG "metric": { 22:49:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:49:34 DEBUG }, 22:49:34 DEBUG "value": [ 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG [...] 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:49:34 DEBUG }, 22:49:34 DEBUG "value": [ 22:49:34 DEBUG 1713649774, 22:49:34 DEBUG "3320.133333333333" 22:49:34 DEBUG ] 22:49:34 DEBUG } 22:49:34 DEBUG ] 22:49:34 DEBUG } 22:49:34 DEBUG } 22:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:34 DEBUG ----- output ----- 22:49:34 DEBUG { 22:49:34 DEBUG "status": "success", 22:49:34 DEBUG "data": { 22:49:34 DEBUG "resultType": "vector", 22:49:34 DEBUG "result": [ 22:49:34 DEBUG { 22:49:34 DEBUG "metric": { 22:49:34 DEBUG "pod": "ds-idrepo-1" 22:49:34 DEBUG }, 22:49:34 DEBUG "value": [ 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG [...] 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG 22:49:34 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:49:34 DEBUG }, 22:49:34 DEBUG "value": [ 22:49:34 DEBUG 1713649774, 22:49:34 DEBUG "0" 22:49:34 DEBUG ] 22:49:34 DEBUG } 22:49:34 DEBUG ] 22:49:34 DEBUG } 22:49:34 DEBUG } 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649775" 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG { 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG "data": { 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG "result": [ 22:49:35 DEBUG { 22:49:35 DEBUG "metric": { 22:49:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG [...] 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG "3328.6666666666665" 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG } 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649775" 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649775" 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649775" 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649775" 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG { 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG "data": { 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG "result": [ 22:49:35 DEBUG { 22:49:35 DEBUG "metric": { 22:49:35 DEBUG "pod": "ds-idrepo-0" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG [...] 22:49:35 DEBUG 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:35 DEBUG 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG { 22:49:35 DEBUG "pod": "ds-idrepo-2" 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG }, 22:49:35 DEBUG "data": { 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:49:35 DEBUG "value": [ 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG "result": [ 22:49:35 DEBUG { 22:49:35 DEBUG "0" 22:49:35 DEBUG { 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG ] 22:49:35 DEBUG "metric": { 22:49:35 DEBUG "data": { 22:49:35 DEBUG } 22:49:35 DEBUG "instance": "10.142.0.107:9100" 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG ] 22:49:35 DEBUG }, 22:49:35 DEBUG "result": [ 22:49:35 DEBUG } 22:49:35 DEBUG "value": [ 22:49:35 DEBUG { 22:49:35 DEBUG } 22:49:35 DEBUG 22:49:35 DEBUG "metric": { 22:49:35 DEBUG 22:49:35 DEBUG "pod": "ds-cts-0" 22:49:35 DEBUG 22:49:35 DEBUG }, 22:49:35 DEBUG [...] 22:49:35 DEBUG "value": [ 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG "instance": "10.142.0.43:9100" 22:49:35 DEBUG [...] 22:49:35 DEBUG }, 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:49:35 DEBUG 22:49:35 DEBUG "value": [ 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG { 22:49:35 DEBUG 22:49:35 DEBUG "0.42519999999999336" 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG "pod": "ds-cts-2" 22:49:35 DEBUG ] 22:49:35 DEBUG "data": { 22:49:35 DEBUG }, 22:49:35 DEBUG } 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG "value": [ 22:49:35 DEBUG ] 22:49:35 DEBUG "result": [ 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG } 22:49:35 DEBUG { 22:49:35 DEBUG "0" 22:49:35 DEBUG } 22:49:35 DEBUG "metric": { 22:49:35 DEBUG ] 22:49:35 DEBUG "pod": "ds-cts-0" 22:49:35 DEBUG } 22:49:35 DEBUG }, 22:49:35 DEBUG ] 22:49:35 DEBUG "value": [ 22:49:35 DEBUG } 22:49:35 DEBUG 22:49:35 DEBUG } 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG [...] 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG "pod": "ds-cts-2" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG "31.133333333333333" 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG } 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649775" 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649775" 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG { 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG "data": { 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG "result": [ 22:49:35 DEBUG { 22:49:35 DEBUG "metric": { 22:49:35 DEBUG "pod": "ds-idrepo-0" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG [...] 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG "pod": "ds-idrepo-2" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG "0" 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG } 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG { 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG "data": { 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG "result": [ 22:49:35 DEBUG { 22:49:35 DEBUG "metric": { 22:49:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG [...] 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649775" 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG "600240" 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG } 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649775" 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649775" 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649775" 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649775" 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG { 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG "data": { 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG "result": [ 22:49:35 DEBUG { 22:49:35 DEBUG "metric": { 22:49:35 DEBUG "instance": "10.142.0.107:9100" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG [...] 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG "instance": "10.142.0.43:9100" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG "100274909.63333333" 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG } 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG { 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG "data": { 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG "result": [ 22:49:35 DEBUG { 22:49:35 DEBUG "metric": { 22:49:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG [...] 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG "3328.0666666666666" 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG } 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG --- http response --- 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG { 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG "data": { 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG "result": [ 22:49:35 DEBUG { 22:49:35 DEBUG "metric": { 22:49:35 DEBUG "pod": "ds-idrepo-1" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG [...] 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG "pod": "ds-idrepo-2" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG "6.8387663737648445" 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG } 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG { 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG "data": { 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG "result": [ 22:49:35 DEBUG { 22:49:35 DEBUG "metric": { 22:49:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG [...] 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG "11.658427177454161" 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG } 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG { 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG "data": { 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG "result": [ 22:49:35 DEBUG { 22:49:35 DEBUG "metric": { 22:49:35 DEBUG "pod": "ds-idrepo-1" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG [...] 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG "pod": "ds-idrepo-2" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG "2852.7320200148724" 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG } 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649775" 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649775" 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG { 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG "data": { 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG "result": [ 22:49:35 DEBUG { 22:49:35 DEBUG "metric": { 22:49:35 DEBUG "instance": "10.142.0.107:9100" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG [...] 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG "instance": "10.142.0.43:9100" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG "4204707496960" 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG } 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG { 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG "data": { 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG "result": [ 22:49:35 DEBUG { 22:49:35 DEBUG "metric": { 22:49:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG [...] 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG "101.875" 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG } 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649775" 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG { 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG "data": { 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG "result": [ 22:49:35 DEBUG { 22:49:35 DEBUG "metric": { 22:49:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG [...] 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG "3333.5" 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG } 22:49:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649775" 22:49:35 INFO [http_cmd]: http status code OK 22:49:35 DEBUG --- status code --- 22:49:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:49:35 DEBUG --- http response --- 22:49:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:49:35 DEBUG ----- output ----- 22:49:35 DEBUG { 22:49:35 DEBUG "status": "success", 22:49:35 DEBUG "data": { 22:49:35 DEBUG "resultType": "vector", 22:49:35 DEBUG "result": [ 22:49:35 DEBUG { 22:49:35 DEBUG "metric": { 22:49:35 DEBUG "instance": "10.142.0.107:9100" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG [...] 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG 22:49:35 DEBUG "instance": "10.142.0.43:9100" 22:49:35 DEBUG }, 22:49:35 DEBUG "value": [ 22:49:35 DEBUG 1713649775, 22:49:35 DEBUG "100277107.46666667" 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG ] 22:49:35 DEBUG } 22:49:35 DEBUG } 22:49:35 INFO 22:49:35 INFO [loop_until]: kubectl --namespace=xlou top pods 22:49:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:49:35 INFO [loop_until]: OK (rc = 0) 22:49:35 DEBUG --- stdout --- 22:49:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22506m 8777Mi am-767f6b67b5-qcqz2 22573m 8900Mi am-767f6b67b5-vbltz 23723m 8942Mi ds-cts-0 19822m 12291Mi ds-cts-1 20118m 12393Mi ds-cts-2 20935m 12542Mi ds-idrepo-0 9727m 3930Mi ds-idrepo-1 9374m 3956Mi ds-idrepo-2 9715m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11801m 3391Mi 22:49:35 DEBUG --- stderr --- 22:49:35 DEBUG 22:49:35 INFO 22:49:35 INFO [loop_until]: kubectl --namespace=xlou top node 22:49:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:49:35 INFO [loop_until]: OK (rc = 0) 22:49:35 DEBUG --- stdout --- 22:49:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6276Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9995m 33% 7692Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21921m 73% 11518Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3387Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19788m 66% 15540Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5196Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22384m 74% 11807Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19649m 65% 3126Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24811m 83% 11200Mi 10% gke-xlou-cdm-ds-0f601751-j488 10319m 34% 7288Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19736m 66% 17423Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10064m 33% 7937Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20537m 68% 15249Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11952m 75% 11277Mi 19% 22:49:35 DEBUG --- stderr --- 22:49:35 DEBUG 22:50:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649834" 22:50:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649834" 22:50:34 ERROR [http_cmd]: ERROR 22:50:34 INFO [http_cmd]: http status code OK 22:50:34 ERROR --- status code --- 22:50:34 DEBUG --- status code --- 22:50:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:50:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:34 ERROR --- http response --- 22:50:34 DEBUG --- http response --- 22:50:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:50:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:50:34 DEBUG ----- output ----- 22:50:34 DEBUG { 22:50:34 DEBUG "status": "success", 22:50:34 DEBUG "data": { 22:50:34 DEBUG "resultType": "vector", 22:50:34 DEBUG "result": [ 22:50:34 DEBUG { 22:50:34 DEBUG "metric": { 22:50:34 DEBUG "pod": "am-767f6b67b5-9zw22" 22:50:34 DEBUG }, 22:50:34 DEBUG "value": [ 22:50:34 DEBUG 22:50:34 DEBUG 22:50:34 DEBUG 22:50:34 DEBUG [...] 22:50:34 DEBUG 22:50:34 DEBUG 22:50:34 DEBUG 22:50:34 DEBUG "pod": "am-767f6b67b5-vbltz" 22:50:34 DEBUG }, 22:50:34 DEBUG "value": [ 22:50:34 DEBUG 1713649834, 22:50:34 DEBUG "0" 22:50:34 DEBUG ] 22:50:34 DEBUG } 22:50:34 DEBUG ] 22:50:34 DEBUG } 22:50:34 DEBUG } 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG { 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:50:35 DEBUG "result": [ 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "pod": "ds-idrepo-0" 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG "metric": { 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG "pod": "ds-idrepo-0" 22:50:35 DEBUG }, 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG "pod": "ds-idrepo-0" 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG }, 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG "value": [ 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG { 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG [...] 22:50:35 DEBUG { 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "data": { 22:50:35 DEBUG 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:35 DEBUG "data": { 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "pod": "ds-idrepo-2" 22:50:35 DEBUG 22:50:35 DEBUG "result": [ 22:50:35 DEBUG "pod": "ds-cts-2" 22:50:35 DEBUG { 22:50:35 DEBUG "result": [ 22:50:35 DEBUG }, 22:50:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 22:50:35 DEBUG { 22:50:35 DEBUG }, 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG { 22:50:35 DEBUG "value": [ 22:50:35 DEBUG }, 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "value": [ 22:50:35 DEBUG "data": { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG "pod": "ds-idrepo-0" 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "instance": "10.142.0.107:9100" 22:50:35 DEBUG "0" 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG }, 22:50:35 DEBUG "0" 22:50:35 DEBUG "result": [ 22:50:35 DEBUG }, 22:50:35 DEBUG ] 22:50:35 DEBUG "0" 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG "value": [ 22:50:35 DEBUG ] 22:50:35 DEBUG { 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG 22:50:35 DEBUG } 22:50:35 DEBUG "metric": { 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 DEBUG 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG 22:50:35 DEBUG ] 22:50:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:50:35 DEBUG 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG 22:50:35 DEBUG } 22:50:35 DEBUG }, 22:50:35 DEBUG [...] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 DEBUG [...] 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:35 DEBUG } 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG } 22:50:35 DEBUG 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG { 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG 22:50:35 DEBUG "instance": "10.142.0.43:9100" 22:50:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:50:35 DEBUG "data": { 22:50:35 DEBUG [...] 22:50:35 DEBUG }, 22:50:35 DEBUG }, 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG 22:50:35 DEBUG "value": [ 22:50:35 DEBUG "value": [ 22:50:35 DEBUG "result": [ 22:50:35 DEBUG 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG { 22:50:35 DEBUG 22:50:35 DEBUG "0.07100000000000743" 22:50:35 DEBUG "0" 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:50:35 DEBUG ] 22:50:35 DEBUG ] 22:50:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:50:35 DEBUG }, 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG ] 22:50:35 DEBUG ] 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 DEBUG 22:50:35 DEBUG "3656007680" 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 DEBUG 22:50:35 DEBUG ] 22:50:35 DEBUG 22:50:35 DEBUG } 22:50:35 DEBUG [...] 22:50:35 DEBUG ] 22:50:35 DEBUG 22:50:35 DEBUG } 22:50:35 DEBUG 22:50:35 DEBUG } 22:50:35 DEBUG 22:50:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "3332.5666666666666" 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "pod": "ds-cts-0" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "pod": "ds-cts-2" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "0.48123333333333373" 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "3331.4" 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "pod": "ds-idrepo-0" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "pod": "ds-idrepo-2" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "0" 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "pod": "ds-idrepo-0" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "pod": "ds-idrepo-2" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "0" 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "pod": "ds-cts-0" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "pod": "ds-cts-2" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "26.5" 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "pod": "ds-cts-0" 22:50:35 DEBUG "instance": "10.142.0.107:9100" 22:50:35 DEBUG }, 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "pod": "ds-cts-2" 22:50:35 DEBUG "instance": "10.142.0.43:9100" 22:50:35 DEBUG }, 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "0" 22:50:35 DEBUG "0.3840666666667478" 22:50:35 DEBUG ] 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "600252.5" 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "instance": "10.142.0.107:9100" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "instance": "10.142.0.43:9100" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "100276051.16666666" 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "3332.866666666667" 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "instance": "10.142.0.107:9100" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "instance": "10.142.0.43:9100" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "4209686076416" 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "pod": "ds-idrepo-0" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "474.71508595711805" 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "pod": "ds-idrepo-0" 22:50:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:50:35 DEBUG }, 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:50:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:50:35 DEBUG }, 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "0" 22:50:35 DEBUG "11.665774953024213" 22:50:35 DEBUG ] 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649835" 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "3333.4333333333334" 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "101.875" 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649835" 22:50:35 INFO [http_cmd]: http status code OK 22:50:35 DEBUG --- status code --- 22:50:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:50:35 DEBUG --- http response --- 22:50:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:50:35 DEBUG ----- output ----- 22:50:35 DEBUG { 22:50:35 DEBUG "status": "success", 22:50:35 DEBUG "data": { 22:50:35 DEBUG "resultType": "vector", 22:50:35 DEBUG "result": [ 22:50:35 DEBUG { 22:50:35 DEBUG "metric": { 22:50:35 DEBUG "instance": "10.142.0.107:9100" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG [...] 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG 22:50:35 DEBUG "instance": "10.142.0.43:9100" 22:50:35 DEBUG }, 22:50:35 DEBUG "value": [ 22:50:35 DEBUG 1713649835, 22:50:35 DEBUG "100277288.9" 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG ] 22:50:35 DEBUG } 22:50:35 DEBUG } 22:50:35 INFO 22:50:35 INFO [loop_until]: kubectl --namespace=xlou top pods 22:50:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:50:35 INFO [loop_until]: OK (rc = 0) 22:50:35 DEBUG --- stdout --- 22:50:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21597m 8779Mi am-767f6b67b5-qcqz2 22079m 8900Mi am-767f6b67b5-vbltz 24433m 8938Mi ds-cts-0 20026m 12292Mi ds-cts-1 19969m 12393Mi ds-cts-2 20653m 12539Mi ds-idrepo-0 9248m 3934Mi ds-idrepo-1 9490m 3955Mi ds-idrepo-2 9520m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11772m 3393Mi 22:50:35 DEBUG --- stderr --- 22:50:35 DEBUG 22:50:35 INFO 22:50:35 INFO [loop_until]: kubectl --namespace=xlou top node 22:50:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:50:36 INFO [loop_until]: OK (rc = 0) 22:50:36 DEBUG --- stdout --- 22:50:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6288Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9960m 33% 7693Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21880m 73% 11516Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 76m 0% 3375Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19703m 65% 15534Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 169m 0% 5191Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22336m 74% 11808Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19673m 65% 3117Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24704m 82% 11194Mi 10% gke-xlou-cdm-ds-0f601751-j488 10296m 34% 7287Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19983m 66% 17421Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10112m 33% 7941Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20689m 69% 15246Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11900m 74% 11279Mi 19% 22:50:36 DEBUG --- stderr --- 22:50:36 DEBUG 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649895" 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649895" 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649895" 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 ERROR [http_cmd]: ERROR 22:51:35 DEBUG --- status code --- 22:51:35 ERROR --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 ERROR --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "0" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:51:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "3662508032" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "instance": "10.142.0.107:9100" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "instance": "10.142.0.43:9100" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "0.07133333333334045" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "ds-idrepo-1" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "59815.43457536773" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "3349.1" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "ds-cts-0" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "ds-cts-2" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "0.5365" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "ds-idrepo-1" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "0" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "ds-idrepo-0" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "ds-cts-2" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "0" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "ds-idrepo-0" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "ds-idrepo-2" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "0" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "3345.366666666667" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "ds-idrepo-0" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "ds-idrepo-2" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "0" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649895" 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "ds-cts-0" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "ds-cts-2" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "0" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649895" 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "ds-idrepo-0" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "ds-idrepo-2" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "0" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "ds-cts-0" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "ds-cts-2" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "30.03333333333333" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "instance": "10.142.0.107:9100" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "instance": "10.142.0.43:9100" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "4214295868416" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "instance": "10.142.0.107:9100" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "instance": "10.142.0.43:9100" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "0.38396666666676443" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "ds-idrepo-1" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "443.0772931508721" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "600023" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649895" 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649895" 22:51:35 DEBUG ----- output ----- 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG { 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG "data": { 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:35 DEBUG 22:51:35 DEBUG ----- output ----- 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:51:35 DEBUG { 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG }, 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG "value": [ 22:51:35 DEBUG "data": { 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "3334.1" 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:35 DEBUG "result": [ 22:51:35 DEBUG ] 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG } 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG ] 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "instance": "10.142.0.107:9100" 22:51:35 DEBUG } 22:51:35 DEBUG "data": { 22:51:35 DEBUG }, 22:51:35 DEBUG } 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "value": [ 22:51:35 DEBUG "result": [ 22:51:35 DEBUG 22:51:35 DEBUG { 22:51:35 DEBUG 22:51:35 DEBUG "metric": { 22:51:35 DEBUG 22:51:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:51:35 DEBUG [...] 22:51:35 DEBUG }, 22:51:35 DEBUG 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "instance": "10.142.0.43:9100" 22:51:35 DEBUG 22:51:35 DEBUG }, 22:51:35 DEBUG [...] 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG 22:51:35 DEBUG "100392583.53333333" 22:51:35 DEBUG 22:51:35 DEBUG ] 22:51:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:51:35 DEBUG } 22:51:35 DEBUG }, 22:51:35 DEBUG ] 22:51:35 DEBUG "value": [ 22:51:35 DEBUG } 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG } 22:51:35 DEBUG "3348.9333333333334" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "ds-idrepo-1" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "11.73887111782375" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "11.769473485034508" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649895" 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "101.875" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649895" 22:51:35 INFO [http_cmd]: http status code OK 22:51:35 DEBUG --- status code --- 22:51:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:51:35 DEBUG --- http response --- 22:51:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:51:35 DEBUG ----- output ----- 22:51:35 DEBUG { 22:51:35 DEBUG "status": "success", 22:51:35 DEBUG "data": { 22:51:35 DEBUG "resultType": "vector", 22:51:35 DEBUG "result": [ 22:51:35 DEBUG { 22:51:35 DEBUG "metric": { 22:51:35 DEBUG "instance": "10.142.0.107:9100" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG [...] 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG 22:51:35 DEBUG "instance": "10.142.0.43:9100" 22:51:35 DEBUG }, 22:51:35 DEBUG "value": [ 22:51:35 DEBUG 1713649895, 22:51:35 DEBUG "100393068.73333333" 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG ] 22:51:35 DEBUG } 22:51:35 DEBUG } 22:51:35 INFO 22:51:35 INFO [loop_until]: kubectl --namespace=xlou top pods 22:51:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:51:35 INFO [loop_until]: OK (rc = 0) 22:51:35 DEBUG --- stdout --- 22:51:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21889m 8776Mi am-767f6b67b5-qcqz2 21767m 8902Mi am-767f6b67b5-vbltz 24785m 8953Mi ds-cts-0 20589m 12295Mi ds-cts-1 20152m 12408Mi ds-cts-2 20749m 12535Mi ds-idrepo-0 9305m 3934Mi ds-idrepo-1 9378m 3955Mi ds-idrepo-2 9671m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 4m 1238Mi lodemon-5497cb87d7-mlld4 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11610m 3399Mi 22:51:35 DEBUG --- stderr --- 22:51:35 DEBUG 22:51:36 INFO 22:51:36 INFO [loop_until]: kubectl --namespace=xlou top node 22:51:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:51:36 INFO [loop_until]: OK (rc = 0) 22:51:36 DEBUG --- stdout --- 22:51:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6276Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9961m 33% 7700Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21923m 73% 11506Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3369Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20049m 67% 15529Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5224Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22383m 74% 11820Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19748m 66% 3132Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24712m 82% 11219Mi 10% gke-xlou-cdm-ds-0f601751-j488 10229m 34% 7286Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19723m 66% 17425Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10102m 33% 7935Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20337m 68% 15238Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 12015m 75% 11282Mi 19% 22:51:36 DEBUG --- stderr --- 22:51:36 DEBUG 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "19197.11581793601" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713649955" 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649955" 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713649955" 22:52:35 ERROR [http_cmd]: ERROR 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 ERROR --- status code --- 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- status code --- 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 ERROR --- http response --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:52:35 DEBUG { 22:52:35 DEBUG { 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG { 22:52:35 DEBUG "data": { 22:52:35 DEBUG "data": { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "result": [ 22:52:35 DEBUG "result": [ 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG { 22:52:35 DEBUG { 22:52:35 DEBUG "result": [ 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG { 22:52:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:52:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:52:35 DEBUG "metric": { 22:52:35 DEBUG }, 22:52:35 DEBUG }, 22:52:35 DEBUG "pod": "ds-cts-0" 22:52:35 DEBUG "value": [ 22:52:35 DEBUG "value": [ 22:52:35 DEBUG }, 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:52:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:52:35 DEBUG 22:52:35 DEBUG }, 22:52:35 DEBUG }, 22:52:35 DEBUG "pod": "ds-cts-2" 22:52:35 DEBUG "value": [ 22:52:35 DEBUG "value": [ 22:52:35 DEBUG }, 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG "0" 22:52:35 DEBUG "0" 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG ] 22:52:35 DEBUG ] 22:52:35 DEBUG "0.4893333333333317" 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG ] 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "instance": "10.142.0.107:9100" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "instance": "10.142.0.43:9100" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "0.07500000000000379" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "3336.7" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "3665420288" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "3327.2999999999997" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "pod": "ds-idrepo-0" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "pod": "ds-idrepo-2" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "0" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "pod": "ds-idrepo-0" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "pod": "ds-cts-2" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "0" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "pod": "ds-cts-0" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "pod": "ds-cts-2" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "0" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "pod": "ds-idrepo-0" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "pod": "ds-idrepo-2" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "0" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649955" 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649955" 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "pod": "ds-idrepo-0" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 DEBUG 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713649955" 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG "pod": "ds-idrepo-2" 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG }, 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:52:35 DEBUG "value": [ 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "0" 22:52:35 DEBUG { 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG ] 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG "data": { 22:52:35 DEBUG } 22:52:35 DEBUG "data": { 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG ] 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG "result": [ 22:52:35 DEBUG } 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:35 DEBUG } 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "pod": "ds-cts-0" 22:52:35 DEBUG { 22:52:35 DEBUG "instance": "10.142.0.107:9100" 22:52:35 DEBUG }, 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG "data": { 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "result": [ 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG { 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG "metric": { 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 DEBUG [...] 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:35 DEBUG 22:52:35 DEBUG "instance": "10.142.0.107:9100" 22:52:35 DEBUG 22:52:35 DEBUG ----- output ----- 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG 22:52:35 DEBUG }, 22:52:35 DEBUG 22:52:35 DEBUG { 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG "pod": "ds-cts-2" 22:52:35 DEBUG 22:52:35 DEBUG "instance": "10.142.0.43:9100" 22:52:35 DEBUG "data": { 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG }, 22:52:35 DEBUG 22:52:35 DEBUG }, 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "value": [ 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:35 DEBUG 22:52:35 DEBUG "value": [ 22:52:35 DEBUG "result": [ 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG [...] 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG { 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG "30.366666666666667" 22:52:35 DEBUG { 22:52:35 DEBUG 22:52:35 DEBUG "4219610905600" 22:52:35 DEBUG "metric": { 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG ] 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG 22:52:35 DEBUG ] 22:52:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG } 22:52:35 DEBUG "data": { 22:52:35 DEBUG 22:52:35 DEBUG } 22:52:35 DEBUG }, 22:52:35 DEBUG ] 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "instance": "10.142.0.43:9100" 22:52:35 DEBUG ] 22:52:35 DEBUG "value": [ 22:52:35 DEBUG } 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG "result": [ 22:52:35 DEBUG }, 22:52:35 DEBUG } 22:52:35 DEBUG 22:52:35 DEBUG } 22:52:35 DEBUG { 22:52:35 DEBUG { 22:52:35 DEBUG "value": [ 22:52:35 DEBUG } 22:52:35 DEBUG 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "metric": { 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG 22:52:35 DEBUG "data": { 22:52:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:52:35 DEBUG "0.40180000000007676" 22:52:35 DEBUG [...] 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG }, 22:52:35 DEBUG ] 22:52:35 DEBUG 22:52:35 DEBUG "result": [ 22:52:35 DEBUG "value": [ 22:52:35 DEBUG } 22:52:35 DEBUG 22:52:35 DEBUG { 22:52:35 DEBUG 22:52:35 DEBUG ] 22:52:35 DEBUG 22:52:35 DEBUG "metric": { 22:52:35 DEBUG 22:52:35 DEBUG } 22:52:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:52:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:52:35 DEBUG 22:52:35 DEBUG } 22:52:35 DEBUG }, 22:52:35 DEBUG }, 22:52:35 DEBUG [...] 22:52:35 DEBUG "value": [ 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "599938" 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG ] 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG 22:52:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:52:35 DEBUG } 22:52:35 DEBUG [...] 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:52:35 DEBUG }, 22:52:35 DEBUG ] 22:52:35 DEBUG 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG "value": [ 22:52:35 DEBUG } 22:52:35 DEBUG 22:52:35 DEBUG { 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG } 22:52:35 DEBUG 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "11.672229755612973" 22:52:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:52:35 DEBUG "data": { 22:52:35 DEBUG ] 22:52:35 DEBUG }, 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG } 22:52:35 DEBUG "value": [ 22:52:35 DEBUG "result": [ 22:52:35 DEBUG ] 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG { 22:52:35 DEBUG } 22:52:35 DEBUG "22.126812678382652" 22:52:35 DEBUG "metric": { 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:52:35 DEBUG } 22:52:35 DEBUG }, 22:52:35 DEBUG ] 22:52:35 DEBUG "value": [ 22:52:35 DEBUG } 22:52:35 DEBUG 22:52:35 DEBUG } 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "492.23373892143616" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "3328.3333333333335" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649955" 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "3322.6" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "instance": "10.142.0.107:9100" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "instance": "10.142.0.43:9100" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "100223876.93333334" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713649955" 22:52:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713649955" 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "instance": "10.142.0.107:9100" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "instance": "10.142.0.43:9100" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "100225070.96666667" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO [http_cmd]: http status code OK 22:52:35 DEBUG --- status code --- 22:52:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:52:35 DEBUG --- http response --- 22:52:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:52:35 DEBUG ----- output ----- 22:52:35 DEBUG { 22:52:35 DEBUG "status": "success", 22:52:35 DEBUG "data": { 22:52:35 DEBUG "resultType": "vector", 22:52:35 DEBUG "result": [ 22:52:35 DEBUG { 22:52:35 DEBUG "metric": { 22:52:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG [...] 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG 22:52:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:52:35 DEBUG }, 22:52:35 DEBUG "value": [ 22:52:35 DEBUG 1713649955, 22:52:35 DEBUG "101.875" 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG ] 22:52:35 DEBUG } 22:52:35 DEBUG } 22:52:35 INFO 22:52:35 INFO [loop_until]: kubectl --namespace=xlou top pods 22:52:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:52:35 INFO [loop_until]: OK (rc = 0) 22:52:35 DEBUG --- stdout --- 22:52:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22020m 8779Mi am-767f6b67b5-qcqz2 22220m 8902Mi am-767f6b67b5-vbltz 24495m 8937Mi ds-cts-0 20042m 12368Mi ds-cts-1 20455m 12404Mi ds-cts-2 20929m 12531Mi ds-idrepo-0 9604m 3933Mi ds-idrepo-1 9324m 3955Mi ds-idrepo-2 9544m 3869Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11569m 3401Mi 22:52:35 DEBUG --- stderr --- 22:52:35 DEBUG 22:52:36 INFO 22:52:36 INFO [loop_until]: kubectl --namespace=xlou top node 22:52:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:52:36 INFO [loop_until]: OK (rc = 0) 22:52:36 DEBUG --- stdout --- 22:52:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6281Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9627m 32% 7702Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21894m 73% 11512Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3380Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19984m 66% 15615Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5183Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22293m 74% 11810Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19705m 66% 3132Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24830m 83% 11199Mi 10% gke-xlou-cdm-ds-0f601751-j488 10236m 34% 7278Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19771m 66% 17436Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10047m 33% 7939Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20557m 68% 15247Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11911m 74% 11281Mi 19% 22:52:36 DEBUG --- stderr --- 22:52:36 DEBUG 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650015" 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650015" 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "3338.233333333333" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "pod": "ds-idrepo-0" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "pod": "ds-idrepo-1" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "0" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650015" 22:53:35 ERROR [http_cmd]: ERROR 22:53:35 ERROR --- status code --- 22:53:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:53:35 ERROR --- http response --- 22:53:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:53:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "instance": "10.142.0.107:9100" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "instance": "10.142.0.43:9100" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "0.07266666666664795" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "3667689472" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "pod": "ds-idrepo-0" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "0" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "pod": "ds-cts-0" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "pod": "ds-cts-2" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "0.5048666666666652" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "0" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "3332.2666666666664" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "pod": "ds-cts-0" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "pod": "ds-cts-2" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "0" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "pod": "ds-idrepo-0" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "pod": "ds-idrepo-2" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "0" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "pod": "ds-idrepo-0" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "pod": "ds-cts-2" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "0" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "pod": "ds-idrepo-0" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "pod": "ds-idrepo-2" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "0" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "599933" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650015" 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650015" 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650015" 22:53:35 DEBUG --- status code --- 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:35 DEBUG { 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG ----- output ----- 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG { 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG "data": { 22:53:35 DEBUG { 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "status": "success", 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG "data": { 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG "result": [ 22:53:35 DEBUG "data": { 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:35 DEBUG "resultType": "vector", 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG "result": [ 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "pod": "ds-idrepo-0" 22:53:35 DEBUG { 22:53:35 DEBUG { 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG { 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:53:35 DEBUG }, 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "metric": { 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG "data": { 22:53:35 DEBUG "pod": "ds-cts-0" 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:35 DEBUG "data": { 22:53:35 DEBUG { 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG }, 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "result": [ 22:53:35 DEBUG "value": [ 22:53:35 DEBUG { 22:53:35 DEBUG { 22:53:35 DEBUG { 22:53:35 DEBUG "result": [ 22:53:35 DEBUG "data": { 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG { 22:53:35 DEBUG 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG "metric": { 22:53:35 DEBUG 22:53:35 DEBUG "data": { 22:53:35 DEBUG "data": { 22:53:35 DEBUG "data": { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "result": [ 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG "instance": "10.142.0.107:9100" 22:53:35 DEBUG 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "instance": "10.142.0.107:9100" 22:53:35 DEBUG { 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG }, 22:53:35 DEBUG [...] 22:53:35 DEBUG "result": [ 22:53:35 DEBUG "result": [ 22:53:35 DEBUG "result": [ 22:53:35 DEBUG }, 22:53:35 DEBUG "metric": { 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG { 22:53:35 DEBUG { 22:53:35 DEBUG { 22:53:35 DEBUG "value": [ 22:53:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:53:35 DEBUG 22:53:35 DEBUG "pod": "ds-idrepo-2" 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG 22:53:35 DEBUG }, 22:53:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:53:35 DEBUG }, 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:53:35 DEBUG "pod": "ds-idrepo-0" 22:53:35 DEBUG "pod": "ds-idrepo-0" 22:53:35 DEBUG 22:53:35 DEBUG "value": [ 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG "pod": "ds-cts-2" 22:53:35 DEBUG }, 22:53:35 DEBUG }, 22:53:35 DEBUG }, 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG [...] 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG "value": [ 22:53:35 DEBUG "value": [ 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "0" 22:53:35 DEBUG 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "3331.9333333333334" 22:53:35 DEBUG ] 22:53:35 DEBUG 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG 22:53:35 DEBUG "31.433333333333334" 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG ] 22:53:35 DEBUG ] 22:53:35 DEBUG [...] 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "instance": "10.142.0.43:9100" 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "instance": "10.142.0.43:9100" 22:53:35 DEBUG 22:53:35 DEBUG }, 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG }, 22:53:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:53:35 DEBUG "value": [ 22:53:35 DEBUG } 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "pod": "ds-idrepo-1" 22:53:35 DEBUG "value": [ 22:53:35 DEBUG }, 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG } 22:53:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:53:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:53:35 DEBUG }, 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG "4225001726976" 22:53:35 DEBUG }, 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG "0.40246666666664815" 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG ] 22:53:35 DEBUG "value": [ 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG ] 22:53:35 DEBUG "11.710641188955671" 22:53:35 DEBUG } 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "3664.29029665858" 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG ] 22:53:35 DEBUG "3333.9666666666667" 22:53:35 DEBUG "0" 22:53:35 DEBUG ] 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650015" 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "instance": "10.142.0.107:9100" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "instance": "10.142.0.43:9100" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "100352500.3" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650015" 22:53:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650015" 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "101.875" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:35 INFO [http_cmd]: http status code OK 22:53:35 DEBUG --- status code --- 22:53:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:53:35 DEBUG --- http response --- 22:53:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:53:35 DEBUG ----- output ----- 22:53:35 DEBUG { 22:53:35 DEBUG "status": "success", 22:53:35 DEBUG "data": { 22:53:35 DEBUG "resultType": "vector", 22:53:35 DEBUG "result": [ 22:53:35 DEBUG { 22:53:35 DEBUG "metric": { 22:53:35 DEBUG "instance": "10.142.0.107:9100" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG [...] 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG 22:53:35 DEBUG "instance": "10.142.0.43:9100" 22:53:35 DEBUG }, 22:53:35 DEBUG "value": [ 22:53:35 DEBUG 1713650015, 22:53:35 DEBUG "100353759.83333333" 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG ] 22:53:35 DEBUG } 22:53:35 DEBUG } 22:53:36 INFO 22:53:36 INFO [loop_until]: kubectl --namespace=xlou top pods 22:53:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:53:36 INFO [loop_until]: OK (rc = 0) 22:53:36 DEBUG --- stdout --- 22:53:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21382m 8781Mi am-767f6b67b5-qcqz2 22213m 8902Mi am-767f6b67b5-vbltz 23980m 8938Mi ds-cts-0 20381m 12369Mi ds-cts-1 19735m 12403Mi ds-cts-2 20175m 12528Mi ds-idrepo-0 9203m 3934Mi ds-idrepo-1 9677m 3955Mi ds-idrepo-2 9651m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11559m 3404Mi 22:53:36 DEBUG --- stderr --- 22:53:36 DEBUG 22:53:36 INFO 22:53:36 INFO [loop_until]: kubectl --namespace=xlou top node 22:53:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:53:36 INFO [loop_until]: OK (rc = 0) 22:53:36 DEBUG --- stdout --- 22:53:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6280Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9954m 33% 7673Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21865m 73% 11512Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3388Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20050m 67% 15612Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 167m 0% 5185Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22340m 74% 11807Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19698m 65% 3119Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 23859m 79% 11194Mi 10% gke-xlou-cdm-ds-0f601751-j488 10371m 34% 7288Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19702m 65% 17426Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10023m 33% 7940Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20628m 69% 15251Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11548m 72% 11279Mi 19% 22:53:36 DEBUG --- stderr --- 22:53:36 DEBUG 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650075" 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650075" 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 ERROR [http_cmd]: ERROR 22:54:35 ERROR --- status code --- 22:54:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:54:35 ERROR --- http response --- 22:54:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:54:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650075" 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (89 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "ds-idrepo-0" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG 22:54:35 DEBUG --- status code --- 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 DEBUG 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [...] 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:35 DEBUG 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG 22:54:35 DEBUG { 22:54:35 DEBUG 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:54:35 DEBUG "data": { 22:54:35 DEBUG }, 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "value": [ 22:54:35 DEBUG "result": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG { 22:54:35 DEBUG "0" 22:54:35 DEBUG "metric": { 22:54:35 DEBUG ] 22:54:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:54:35 DEBUG } 22:54:35 DEBUG }, 22:54:35 DEBUG ] 22:54:35 DEBUG "value": [ 22:54:35 DEBUG } 22:54:35 DEBUG 22:54:35 DEBUG } 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "3670618112" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "ds-cts-0" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "pod": "ds-cts-2" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "0.4943666666666663" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "instance": "10.142.0.107:9100" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "instance": "10.142.0.43:9100" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "0.08233333333333613" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "3368.3333333333335" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "3344.2" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "ds-idrepo-0" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "pod": "ds-idrepo-2" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "0" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "ds-idrepo-0" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "0" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "0" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "ds-cts-0" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "pod": "ds-cts-2" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "0" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "ds-idrepo-0" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "pod": "ds-cts-2" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "0" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "ds-idrepo-0" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "pod": "ds-idrepo-2" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "0" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "599827" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "ds-idrepo-0" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "0.000006346246246236058" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (89 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "ds-idrepo-0" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "0" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650075" 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650075" 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650075" 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "3345.233333333333" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:54:35 DEBUG { 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG { 22:54:35 DEBUG "data": { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "result": [ 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG { 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "result": [ 22:54:35 DEBUG "metric": { 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650075" 22:54:35 DEBUG { 22:54:35 DEBUG "instance": "10.142.0.107:9100" 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "ds-idrepo-0" 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG "metric": { 22:54:35 DEBUG }, 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG "instance": "10.142.0.107:9100" 22:54:35 DEBUG }, 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG "pod": "ds-cts-0" 22:54:35 DEBUG "value": [ 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG }, 22:54:35 DEBUG 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG "value": [ 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:35 DEBUG 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG { 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG { 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG "data": { 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "data": { 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "result": [ 22:54:35 DEBUG "instance": "10.142.0.43:9100" 22:54:35 DEBUG 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "pod": "ds-idrepo-2" 22:54:35 DEBUG 22:54:35 DEBUG { 22:54:35 DEBUG }, 22:54:35 DEBUG "instance": "10.142.0.43:9100" 22:54:35 DEBUG "result": [ 22:54:35 DEBUG }, 22:54:35 DEBUG "pod": "ds-cts-2" 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "value": [ 22:54:35 DEBUG }, 22:54:35 DEBUG { 22:54:35 DEBUG "value": [ 22:54:35 DEBUG }, 22:54:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG "metric": { 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG }, 22:54:35 DEBUG "4230098508800" 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:54:35 DEBUG "0" 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG ] 22:54:35 DEBUG "0.4078666666667601" 22:54:35 DEBUG }, 22:54:35 DEBUG ] 22:54:35 DEBUG "28.4" 22:54:35 DEBUG 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG "value": [ 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG [...] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 DEBUG 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 DEBUG 22:54:35 DEBUG } 22:54:35 DEBUG [...] 22:54:35 DEBUG } 22:54:35 DEBUG 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG 22:54:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG 22:54:35 DEBUG }, 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:54:35 DEBUG "value": [ 22:54:35 DEBUG }, 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG "3333.4666666666667" 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG { 22:54:35 DEBUG ] 22:54:35 DEBUG "11.65153112178794" 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG "data": { 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG "result": [ 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 DEBUG { 22:54:35 DEBUG } 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "instance": "10.142.0.107:9100" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "instance": "10.142.0.43:9100" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "100098748.16666667" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650075" 22:54:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650075" 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "instance": "10.142.0.107:9100" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "instance": "10.142.0.43:9100" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "100099971.6" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:35 INFO [http_cmd]: http status code OK 22:54:35 DEBUG --- status code --- 22:54:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:54:35 DEBUG --- http response --- 22:54:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:54:35 DEBUG ----- output ----- 22:54:35 DEBUG { 22:54:35 DEBUG "status": "success", 22:54:35 DEBUG "data": { 22:54:35 DEBUG "resultType": "vector", 22:54:35 DEBUG "result": [ 22:54:35 DEBUG { 22:54:35 DEBUG "metric": { 22:54:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG [...] 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG 22:54:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:54:35 DEBUG }, 22:54:35 DEBUG "value": [ 22:54:35 DEBUG 1713650075, 22:54:35 DEBUG "101.875" 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG ] 22:54:35 DEBUG } 22:54:35 DEBUG } 22:54:36 INFO 22:54:36 INFO [loop_until]: kubectl --namespace=xlou top pods 22:54:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:54:36 INFO [loop_until]: OK (rc = 0) 22:54:36 DEBUG --- stdout --- 22:54:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21788m 8778Mi am-767f6b67b5-qcqz2 21826m 8901Mi am-767f6b67b5-vbltz 24883m 8940Mi ds-cts-0 20342m 12371Mi ds-cts-1 20013m 12405Mi ds-cts-2 20017m 12561Mi ds-idrepo-0 9493m 3914Mi ds-idrepo-1 9615m 3955Mi ds-idrepo-2 9654m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11795m 3406Mi 22:54:36 DEBUG --- stderr --- 22:54:36 DEBUG 22:54:36 INFO 22:54:36 INFO [loop_until]: kubectl --namespace=xlou top node 22:54:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:54:36 INFO [loop_until]: OK (rc = 0) 22:54:36 DEBUG --- stdout --- 22:54:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6284Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10012m 33% 7675Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21891m 73% 11522Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3388Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19881m 66% 15577Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5196Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22307m 74% 11816Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19663m 65% 3131Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24721m 82% 11190Mi 10% gke-xlou-cdm-ds-0f601751-j488 10301m 34% 7289Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19715m 66% 17428Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10101m 33% 7930Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20610m 69% 15276Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11931m 75% 11282Mi 19% 22:54:36 DEBUG --- stderr --- 22:54:36 DEBUG 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650135" 22:55:35 ERROR [http_cmd]: ERROR 22:55:35 ERROR --- status code --- 22:55:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:55:35 ERROR --- http response --- 22:55:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:55:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650135" 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "instance": "10.142.0.107:9100" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "instance": "10.142.0.43:9100" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "0.07500000000000757" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "ds-cts-0" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "ds-cts-2" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "0.5317999999999984" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "ds-idrepo-0" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "ds-idrepo-2" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "0" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650135" 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "ds-idrepo-0" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "overseer-0-967d957f7-zk66v" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "0" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "3673608192" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "3332.7666666666664" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "3308.5333333333333" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:55:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "ds-cts-0" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "ds-cts-2" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "0" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "ds-idrepo-0" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "0" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "0" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "ds-idrepo-0" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "ds-cts-2" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "0" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "599968" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "ds-idrepo-0" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "ds-idrepo-2" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "0" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650135" 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "ds-idrepo-0" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "0" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG { 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "ds-cts-0" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "ds-cts-2" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "30.7" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "ds-idrepo-0" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "overseer-0-967d957f7-zk66v" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "0" 22:55:35 DEBUG ] 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG } 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG ] 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG } 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG } 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "11.657093948487907" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650135" 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "instance": "10.142.0.107:9100" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "instance": "10.142.0.43:9100" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "4235238218752" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "3334.233333333333" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650135" 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650135" 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "ds-idrepo-0" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "ds-idrepo-2" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "0" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG { 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG "data": { 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG "result": [ 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG { 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "instance": "10.142.0.107:9100" 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG }, 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG "value": [ 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "instance": "10.142.0.107:9100" 22:55:35 DEBUG 22:55:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:55:35 DEBUG }, 22:55:35 DEBUG [...] 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "instance": "10.142.0.43:9100" 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG }, 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "100437785.63333333" 22:55:35 DEBUG 22:55:35 DEBUG "instance": "10.142.0.43:9100" 22:55:35 DEBUG ] 22:55:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:55:35 DEBUG }, 22:55:35 DEBUG } 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG ] 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG } 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "0.40640000000002297" 22:55:35 DEBUG } 22:55:35 DEBUG "3341.4666666666667" 22:55:35 DEBUG ] 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650135" 22:55:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650135" 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "101.875" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:35 INFO [http_cmd]: http status code OK 22:55:35 DEBUG --- status code --- 22:55:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:55:35 DEBUG --- http response --- 22:55:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:55:35 DEBUG ----- output ----- 22:55:35 DEBUG { 22:55:35 DEBUG "status": "success", 22:55:35 DEBUG "data": { 22:55:35 DEBUG "resultType": "vector", 22:55:35 DEBUG "result": [ 22:55:35 DEBUG { 22:55:35 DEBUG "metric": { 22:55:35 DEBUG "instance": "10.142.0.107:9100" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG [...] 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG 22:55:35 DEBUG "instance": "10.142.0.43:9100" 22:55:35 DEBUG }, 22:55:35 DEBUG "value": [ 22:55:35 DEBUG 1713650135, 22:55:35 DEBUG "100438123.93333334" 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG ] 22:55:35 DEBUG } 22:55:35 DEBUG } 22:55:36 INFO 22:55:36 INFO [loop_until]: kubectl --namespace=xlou top pods 22:55:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:55:36 INFO [loop_until]: OK (rc = 0) 22:55:36 DEBUG --- stdout --- 22:55:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22440m 8779Mi am-767f6b67b5-qcqz2 22184m 8901Mi am-767f6b67b5-vbltz 24707m 8953Mi ds-cts-0 20196m 12365Mi ds-cts-1 19407m 12407Mi ds-cts-2 20051m 12560Mi ds-idrepo-0 9609m 3914Mi ds-idrepo-1 9508m 3947Mi ds-idrepo-2 9795m 3869Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11779m 3409Mi 22:55:36 DEBUG --- stderr --- 22:55:36 DEBUG 22:55:36 INFO 22:55:36 INFO [loop_until]: kubectl --namespace=xlou top node 22:55:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:55:36 INFO [loop_until]: OK (rc = 0) 22:55:36 DEBUG --- stdout --- 22:55:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6293Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9932m 33% 7692Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21885m 73% 11512Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3379Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19966m 66% 15602Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5220Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22353m 74% 11812Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19748m 66% 3125Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24891m 83% 11214Mi 10% gke-xlou-cdm-ds-0f601751-j488 10293m 34% 7289Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19572m 65% 17425Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10037m 33% 7934Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20551m 68% 15274Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11972m 75% 11281Mi 19% 22:55:36 DEBUG --- stderr --- 22:55:36 DEBUG 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650195" 22:56:35 ERROR [http_cmd]: ERROR 22:56:35 ERROR --- status code --- 22:56:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:56:35 ERROR --- http response --- 22:56:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:56:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650195" 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "ds-cts-0" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "ds-cts-2" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "0.5217333333333348" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "instance": "10.142.0.107:9100" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "instance": "10.142.0.43:9100" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "0.07300000000001697" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "ds-idrepo-0" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "ds-idrepo-2" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "0" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650195" 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "3342.9666666666667" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "ds-idrepo-1" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "0" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "3676168192" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "3337.133333333333" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "ds-cts-0" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "ds-cts-2" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "0" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "ds-idrepo-1" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "112140.06227601199" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "0" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "ds-idrepo-0" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "ds-cts-2" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "0" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "600205" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "ds-idrepo-1" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "22.167619822570394" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "ds-idrepo-1" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "962.5756418541802" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "ds-idrepo-0" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "ds-idrepo-2" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "0" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650195" 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "3337.2666666666664" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "ds-cts-0" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "ds-cts-2" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "32.666666666666664" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "11.729571178877272" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650195" 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "3336.8" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "instance": "10.142.0.107:9100" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "instance": "10.142.0.43:9100" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "4240533611520" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650195" 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "101.875" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650195" 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650195" 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "pod": "ds-idrepo-0" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "pod": "ds-idrepo-2" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "0" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "instance": "10.142.0.107:9100" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "instance": "10.142.0.43:9100" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "99968198.93333334" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "instance": "10.142.0.107:9100" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "instance": "10.142.0.43:9100" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "0.3879666666666803" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650195" 22:56:35 INFO [http_cmd]: http status code OK 22:56:35 DEBUG --- status code --- 22:56:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:56:35 DEBUG --- http response --- 22:56:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:56:35 DEBUG ----- output ----- 22:56:35 DEBUG { 22:56:35 DEBUG "status": "success", 22:56:35 DEBUG "data": { 22:56:35 DEBUG "resultType": "vector", 22:56:35 DEBUG "result": [ 22:56:35 DEBUG { 22:56:35 DEBUG "metric": { 22:56:35 DEBUG "instance": "10.142.0.107:9100" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG [...] 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG 22:56:35 DEBUG "instance": "10.142.0.43:9100" 22:56:35 DEBUG }, 22:56:35 DEBUG "value": [ 22:56:35 DEBUG 1713650195, 22:56:35 DEBUG "99965547.9" 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG ] 22:56:35 DEBUG } 22:56:35 DEBUG } 22:56:36 INFO 22:56:36 INFO [loop_until]: kubectl --namespace=xlou top pods 22:56:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:56:36 INFO [loop_until]: OK (rc = 0) 22:56:36 DEBUG --- stdout --- 22:56:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 21627m 8780Mi am-767f6b67b5-qcqz2 22129m 8904Mi am-767f6b67b5-vbltz 23870m 8939Mi ds-cts-0 19724m 12365Mi ds-cts-1 20327m 12406Mi ds-cts-2 19894m 12557Mi ds-idrepo-0 9605m 3914Mi ds-idrepo-1 9594m 3947Mi ds-idrepo-2 9428m 3869Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 4m 1238Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11724m 3412Mi 22:56:36 DEBUG --- stderr --- 22:56:36 DEBUG 22:56:36 INFO 22:56:36 INFO [loop_until]: kubectl --namespace=xlou top node 22:56:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:56:36 INFO [loop_until]: OK (rc = 0) 22:56:36 DEBUG --- stdout --- 22:56:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6286Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9969m 33% 7692Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21874m 73% 11514Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3389Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20166m 67% 15614Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5214Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22479m 75% 11821Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19681m 65% 3124Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24692m 82% 11188Mi 10% gke-xlou-cdm-ds-0f601751-j488 10259m 34% 7292Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19914m 66% 17428Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10046m 33% 7921Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20515m 68% 15276Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 12006m 75% 11281Mi 19% 22:56:36 DEBUG --- stderr --- 22:56:36 DEBUG 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650255" 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650255" 22:57:35 ERROR [http_cmd]: ERROR 22:57:35 ERROR --- status code --- 22:57:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:57:35 ERROR --- http response --- 22:57:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:57:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "instance": "10.142.0.107:9100" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "instance": "10.142.0.43:9100" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "0.07333333333334621" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "ds-cts-0" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "ds-cts-2" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "0.5275333333333341" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "ds-idrepo-0" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "ds-idrepo-2" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "0" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG { 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG --- http response --- 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:57:35 DEBUG "data": { 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG { 22:57:35 DEBUG { 22:57:35 DEBUG "result": [ 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG { 22:57:35 DEBUG "data": { 22:57:35 DEBUG "data": { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:57:35 DEBUG "result": [ 22:57:35 DEBUG "result": [ 22:57:35 DEBUG }, 22:57:35 DEBUG { 22:57:35 DEBUG { 22:57:35 DEBUG "value": [ 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650255" 22:57:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:57:35 DEBUG "pod": "ds-cts-0" 22:57:35 DEBUG 22:57:35 DEBUG }, 22:57:35 DEBUG }, 22:57:35 DEBUG 22:57:35 DEBUG "value": [ 22:57:35 DEBUG "value": [ 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG [...] 22:57:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG }, 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:57:35 DEBUG "pod": "ds-cts-2" 22:57:35 DEBUG "3321.9666666666667" 22:57:35 DEBUG }, 22:57:35 DEBUG }, 22:57:35 DEBUG ] 22:57:35 DEBUG "value": [ 22:57:35 DEBUG "value": [ 22:57:35 DEBUG } 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG ] 22:57:35 DEBUG "3803.9333333333334" 22:57:35 DEBUG "0" 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "ds-idrepo-0" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "0" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "3678183424" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "ds-idrepo-0" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "304266624.9506679" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "0" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "600088.5" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "ds-idrepo-0" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "ds-cts-2" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "0" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "ds-idrepo-0" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "10.409196877792654" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "ds-idrepo-0" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "533.978152318344" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "ds-idrepo-0" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "ds-idrepo-2" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "0" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "3325.3333333333335" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650255" 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "3325.9666666666667" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650255" 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "ds-cts-0" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "ds-cts-2" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "27.066666666666666" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "11.698696566127847" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "instance": "10.142.0.107:9100" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "instance": "10.142.0.43:9100" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "100858604.56666666" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650255" 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "instance": "10.142.0.107:9100" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "instance": "10.142.0.43:9100" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "4245840212992" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650255" 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "ds-idrepo-0" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "ds-idrepo-2" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "0" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650255" 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "102.8775" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "instance": "10.142.0.107:9100" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "instance": "10.142.0.43:9100" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "0.4037666666666316" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650255" 22:57:35 INFO [http_cmd]: http status code OK 22:57:35 DEBUG --- status code --- 22:57:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:57:35 DEBUG --- http response --- 22:57:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:57:35 DEBUG ----- output ----- 22:57:35 DEBUG { 22:57:35 DEBUG "status": "success", 22:57:35 DEBUG "data": { 22:57:35 DEBUG "resultType": "vector", 22:57:35 DEBUG "result": [ 22:57:35 DEBUG { 22:57:35 DEBUG "metric": { 22:57:35 DEBUG "instance": "10.142.0.107:9100" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG [...] 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG 22:57:35 DEBUG "instance": "10.142.0.43:9100" 22:57:35 DEBUG }, 22:57:35 DEBUG "value": [ 22:57:35 DEBUG 1713650255, 22:57:35 DEBUG "100857832.63333333" 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG ] 22:57:35 DEBUG } 22:57:35 DEBUG } 22:57:36 INFO 22:57:36 INFO [loop_until]: kubectl --namespace=xlou top pods 22:57:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:57:36 INFO [loop_until]: OK (rc = 0) 22:57:36 DEBUG --- stdout --- 22:57:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22873m 8796Mi am-767f6b67b5-qcqz2 22160m 8918Mi am-767f6b67b5-vbltz 24460m 8938Mi ds-cts-0 20226m 12368Mi ds-cts-1 19901m 12403Mi ds-cts-2 20372m 12599Mi ds-idrepo-0 9597m 3914Mi ds-idrepo-1 9238m 3947Mi ds-idrepo-2 9575m 3869Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11770m 3498Mi 22:57:36 DEBUG --- stderr --- 22:57:36 DEBUG 22:57:36 INFO 22:57:36 INFO [loop_until]: kubectl --namespace=xlou top node 22:57:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:57:36 INFO [loop_until]: OK (rc = 0) 22:57:36 DEBUG --- stdout --- 22:57:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6296Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9953m 33% 7690Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22445m 75% 11539Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3373Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20190m 67% 15616Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5208Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22466m 75% 11839Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19770m 66% 3122Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24830m 83% 11186Mi 10% gke-xlou-cdm-ds-0f601751-j488 10235m 34% 7284Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19942m 66% 17425Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 9993m 33% 7920Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20596m 68% 15318Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11559m 72% 11365Mi 19% 22:57:36 DEBUG --- stderr --- 22:57:36 DEBUG 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650315" 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650315" 22:58:35 ERROR [http_cmd]: ERROR 22:58:35 ERROR --- status code --- 22:58:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:58:35 ERROR --- http response --- 22:58:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:58:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "pod": "ds-idrepo-0" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "ds-idrepo-2" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "0" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "instance": "10.142.0.107:9100" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "instance": "10.142.0.43:9100" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "0.08766666666667029" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "pod": "ds-cts-0" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "ds-cts-2" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "0.576333333333334" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG { 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "data": { 22:58:35 DEBUG "pod": "ds-cts-0" 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG }, 22:58:35 DEBUG "result": [ 22:58:35 DEBUG "value": [ 22:58:35 DEBUG { 22:58:35 DEBUG 22:58:35 DEBUG "metric": { 22:58:35 DEBUG 22:58:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:58:35 DEBUG 22:58:35 DEBUG }, 22:58:35 DEBUG [...] 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "ds-cts-2" 22:58:35 DEBUG [...] 22:58:35 DEBUG }, 22:58:35 DEBUG 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG 22:58:35 DEBUG "0" 22:58:35 DEBUG "pod": "ds-idrepo-2" 22:58:35 DEBUG ] 22:58:35 DEBUG }, 22:58:35 DEBUG } 22:58:35 DEBUG "value": [ 22:58:35 DEBUG ] 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG } 22:58:35 DEBUG "0" 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650315" 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "4129.333333333333" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "3325.7" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "3776729088" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "ds-idrepo-2" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "4206.452382670612" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "0" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "599880" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "pod": "ds-idrepo-0" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "ds-cts-2" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "0" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "ds-idrepo-2" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "6.927165347159962" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 DEBUG "status": "success", 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG "data": { 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG "result": [ 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 22:58:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG }, 22:58:35 DEBUG { 22:58:35 DEBUG "value": [ 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG 22:58:35 DEBUG "data": { 22:58:35 DEBUG 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG 22:58:35 DEBUG "result": [ 22:58:35 DEBUG [...] 22:58:35 DEBUG { 22:58:35 DEBUG 22:58:35 DEBUG "metric": { 22:58:35 DEBUG 22:58:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 22:58:35 DEBUG 22:58:35 DEBUG }, 22:58:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:58:35 DEBUG "value": [ 22:58:35 DEBUG }, 22:58:35 DEBUG 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG 22:58:35 DEBUG "3334.2" 22:58:35 DEBUG [...] 22:58:35 DEBUG ] 22:58:35 DEBUG 22:58:35 DEBUG } 22:58:35 DEBUG 22:58:35 DEBUG ] 22:58:35 DEBUG 22:58:35 DEBUG } 22:58:35 DEBUG "pod": "ds-idrepo-2" 22:58:35 DEBUG } 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "2734.194048735897" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "pod": "ds-idrepo-0" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "ds-idrepo-2" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "0" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650315" 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "3325.9" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "instance": "10.142.0.107:9100" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "instance": "10.142.0.43:9100" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "101581512.13333333" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650315" 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "pod": "ds-cts-0" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "ds-cts-2" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "30.766666666666666" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "11.785895393492382" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650315" 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "instance": "10.142.0.107:9100" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "instance": "10.142.0.43:9100" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "4250751929344" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "pod": "ds-idrepo-0" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "ds-idrepo-2" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "0" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650315" 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650315" 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "108.488" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "instance": "10.142.0.107:9100" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "instance": "10.142.0.43:9100" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "0.39860000000001033" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650315" 22:58:35 INFO [http_cmd]: http status code OK 22:58:35 DEBUG --- status code --- 22:58:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:58:35 DEBUG --- http response --- 22:58:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:58:35 DEBUG ----- output ----- 22:58:35 DEBUG { 22:58:35 DEBUG "status": "success", 22:58:35 DEBUG "data": { 22:58:35 DEBUG "resultType": "vector", 22:58:35 DEBUG "result": [ 22:58:35 DEBUG { 22:58:35 DEBUG "metric": { 22:58:35 DEBUG "instance": "10.142.0.107:9100" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG [...] 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG 22:58:35 DEBUG "instance": "10.142.0.43:9100" 22:58:35 DEBUG }, 22:58:35 DEBUG "value": [ 22:58:35 DEBUG 1713650315, 22:58:35 DEBUG "101580882.69999999" 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG ] 22:58:35 DEBUG } 22:58:35 DEBUG } 22:58:36 INFO 22:58:36 INFO [loop_until]: kubectl --namespace=xlou top pods 22:58:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:58:36 INFO [loop_until]: OK (rc = 0) 22:58:36 DEBUG --- stdout --- 22:58:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22839m 8799Mi am-767f6b67b5-qcqz2 23218m 8919Mi am-767f6b67b5-vbltz 25187m 8953Mi ds-cts-0 20695m 12367Mi ds-cts-1 19396m 12400Mi ds-cts-2 20687m 12595Mi ds-idrepo-0 9441m 3914Mi ds-idrepo-1 9531m 3947Mi ds-idrepo-2 9690m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11918m 3611Mi 22:58:36 DEBUG --- stderr --- 22:58:36 DEBUG 22:58:37 INFO 22:58:37 INFO [loop_until]: kubectl --namespace=xlou top node 22:58:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:58:37 INFO [loop_until]: OK (rc = 0) 22:58:37 DEBUG --- stdout --- 22:58:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6293Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9955m 33% 7694Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22349m 74% 11542Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 78m 0% 3376Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20275m 67% 15605Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5204Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22820m 76% 11831Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19822m 66% 3142Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25222m 84% 11201Mi 10% gke-xlou-cdm-ds-0f601751-j488 10242m 34% 7293Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19444m 65% 17420Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10011m 33% 7925Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20611m 69% 15302Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11982m 75% 11467Mi 19% 22:58:37 DEBUG --- stderr --- 22:58:37 DEBUG 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650375" 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 ERROR [http_cmd]: ERROR 22:59:35 ERROR --- status code --- 22:59:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 22:59:35 ERROR --- http response --- 22:59:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 22:59:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650375" 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:59:35 DEBUG "pod": "ds-idrepo-0" 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG }, 22:59:35 DEBUG { 22:59:35 DEBUG "value": [ 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG 22:59:35 DEBUG "data": { 22:59:35 DEBUG 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG 22:59:35 DEBUG "result": [ 22:59:35 DEBUG [...] 22:59:35 DEBUG { 22:59:35 DEBUG 22:59:35 DEBUG "metric": { 22:59:35 DEBUG 22:59:35 DEBUG "pod": "ds-cts-0" 22:59:35 DEBUG 22:59:35 DEBUG }, 22:59:35 DEBUG "pod": "ds-idrepo-2" 22:59:35 DEBUG "value": [ 22:59:35 DEBUG }, 22:59:35 DEBUG 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG 22:59:35 DEBUG "0" 22:59:35 DEBUG [...] 22:59:35 DEBUG ] 22:59:35 DEBUG 22:59:35 DEBUG } 22:59:35 DEBUG 22:59:35 DEBUG ] 22:59:35 DEBUG 22:59:35 DEBUG } 22:59:35 DEBUG "pod": "ds-cts-2" 22:59:35 DEBUG } 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "0.5753333333333349" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "instance": "10.142.0.107:9100" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "instance": "10.142.0.43:9100" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "0.07499999999998863" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650375" 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG { 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG "data": { 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG { 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "metric": { 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG "data": { 22:59:35 DEBUG "pod": "ds-cts-0" 22:59:35 DEBUG { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG }, 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG "value": [ 22:59:35 DEBUG "data": { 22:59:35 DEBUG { 22:59:35 DEBUG 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "metric": { 22:59:35 DEBUG 22:59:35 DEBUG "result": [ 22:59:35 DEBUG "pod": "ds-idrepo-0" 22:59:35 DEBUG 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG { 22:59:35 DEBUG }, 22:59:35 DEBUG [...] 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG }, 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "value": [ 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:35 DEBUG 22:59:35 DEBUG "pod": "ds-cts-2" 22:59:35 DEBUG 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG [...] 22:59:35 DEBUG }, 22:59:35 DEBUG 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG { 22:59:35 DEBUG 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG [...] 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG "data": { 22:59:35 DEBUG 22:59:35 DEBUG "0" 22:59:35 DEBUG 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:59:35 DEBUG ] 22:59:35 DEBUG 22:59:35 DEBUG "result": [ 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:35 DEBUG }, 22:59:35 DEBUG } 22:59:35 DEBUG 22:59:35 DEBUG { 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG "value": [ 22:59:35 DEBUG ] 22:59:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:59:35 DEBUG "metric": { 22:59:35 DEBUG { 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG } 22:59:35 DEBUG }, 22:59:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "0" 22:59:35 DEBUG } 22:59:35 DEBUG "value": [ 22:59:35 DEBUG }, 22:59:35 DEBUG "data": { 22:59:35 DEBUG ] 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG } 22:59:35 DEBUG "3336.0333333333333" 22:59:35 DEBUG 22:59:35 DEBUG "result": [ 22:59:35 DEBUG ] 22:59:35 DEBUG ] 22:59:35 DEBUG 22:59:35 DEBUG { 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 DEBUG 22:59:35 DEBUG "metric": { 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG [...] 22:59:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:59:35 DEBUG } 22:59:35 DEBUG 22:59:35 DEBUG }, 22:59:35 DEBUG } 22:59:35 DEBUG 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:59:35 DEBUG 22:59:35 DEBUG }, 22:59:35 DEBUG 22:59:35 DEBUG "value": [ 22:59:35 DEBUG [...] 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG 22:59:35 DEBUG "3844.7000000000003" 22:59:35 DEBUG 22:59:35 DEBUG ] 22:59:35 DEBUG 22:59:35 DEBUG } 22:59:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:59:35 DEBUG ] 22:59:35 DEBUG }, 22:59:35 DEBUG } 22:59:35 DEBUG "value": [ 22:59:35 DEBUG } 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "3882561536" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "pod": "ds-idrepo-0" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "1333140.4997484486" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "0" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "599710.5" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "pod": "ds-idrepo-0" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "22.675255643129354" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "pod": "ds-idrepo-0" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "pod": "ds-idrepo-2" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "0" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "pod": "ds-idrepo-0" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "pod": "ds-cts-2" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "0" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650375" 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "11.766385771317806" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "3318.6666666666665" 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 22:59:35 DEBUG ] 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG } 22:59:35 DEBUG { 22:59:35 DEBUG ] 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG } 22:59:35 DEBUG "data": { 22:59:35 DEBUG } 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "pod": "ds-idrepo-0" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "pod": "am-767f6b67b5-qcqz2" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "549.522052658058" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650375" 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG { 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "data": { 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG "result": [ 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG { 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:59:35 DEBUG "pod": "ds-cts-0" 22:59:35 DEBUG "instance": "10.142.0.107:9100" 22:59:35 DEBUG }, 22:59:35 DEBUG }, 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG "value": [ 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG [...] 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:59:35 DEBUG "pod": "ds-cts-2" 22:59:35 DEBUG "instance": "10.142.0.43:9100" 22:59:35 DEBUG }, 22:59:35 DEBUG }, 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG "value": [ 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "3334.9666666666667" 22:59:35 DEBUG "30.866666666666667" 22:59:35 DEBUG "101225165.43333334" 22:59:35 DEBUG ] 22:59:35 DEBUG ] 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG ] 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650375" 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650375" 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "instance": "10.142.0.107:9100" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "instance": "10.142.0.43:9100" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "4255882562560" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "pod": "ds-idrepo-0" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "pod": "ds-idrepo-2" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "0" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "instance": "10.142.0.107:9100" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "instance": "10.142.0.43:9100" 22:59:35 DEBUG }, 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650375" 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "101224434.5" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650375" 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "pod": "am-767f6b67b5-9zw22" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "pod": "am-767f6b67b5-vbltz" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "113.7755" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:35 INFO [http_cmd]: http status code OK 22:59:35 DEBUG --- status code --- 22:59:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 22:59:35 DEBUG --- http response --- 22:59:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 22:59:35 DEBUG ----- output ----- 22:59:35 DEBUG { 22:59:35 DEBUG "status": "success", 22:59:35 DEBUG "data": { 22:59:35 DEBUG "resultType": "vector", 22:59:35 DEBUG "result": [ 22:59:35 DEBUG { 22:59:35 DEBUG "metric": { 22:59:35 DEBUG "instance": "10.142.0.107:9100" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG [...] 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG 22:59:35 DEBUG "instance": "10.142.0.43:9100" 22:59:35 DEBUG }, 22:59:35 DEBUG "value": [ 22:59:35 DEBUG 1713650375, 22:59:35 DEBUG "0.3952000000000453" 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG ] 22:59:35 DEBUG } 22:59:35 DEBUG } 22:59:36 INFO 22:59:36 INFO [loop_until]: kubectl --namespace=xlou top pods 22:59:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:59:36 INFO [loop_until]: OK (rc = 0) 22:59:36 DEBUG --- stdout --- 22:59:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 23250m 8799Mi am-767f6b67b5-qcqz2 22428m 8958Mi am-767f6b67b5-vbltz 24538m 9008Mi ds-cts-0 20154m 12365Mi ds-cts-1 20035m 12488Mi ds-cts-2 21192m 12596Mi ds-idrepo-0 9570m 3914Mi ds-idrepo-1 9366m 3947Mi ds-idrepo-2 9591m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-5497cb87d7-mlld4 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11729m 3608Mi 22:59:36 DEBUG --- stderr --- 22:59:36 DEBUG 22:59:37 INFO 22:59:37 INFO [loop_until]: kubectl --namespace=xlou top node 22:59:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 22:59:37 INFO [loop_until]: OK (rc = 0) 22:59:37 DEBUG --- stdout --- 22:59:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6299Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9980m 33% 7675Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22549m 75% 11542Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3378Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20096m 67% 15571Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5200Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22552m 75% 11875Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19708m 66% 3146Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24273m 81% 11247Mi 10% gke-xlou-cdm-ds-0f601751-j488 10280m 34% 7290Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19649m 65% 17486Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10031m 33% 7927Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20678m 69% 15296Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11941m 75% 11462Mi 19% 22:59:37 DEBUG --- stderr --- 22:59:37 DEBUG 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650435" 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650435" 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 ERROR [http_cmd]: ERROR 23:00:35 ERROR --- status code --- 23:00:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:00:35 ERROR --- http response --- 23:00:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:00:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "ds-idrepo-0" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "ds-idrepo-2" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "0" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "instance": "10.142.0.107:9100" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "instance": "10.142.0.43:9100" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "0.08666666666666932" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "ds-cts-0" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "ds-cts-2" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "0.4833666666666659" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "3331.1" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "ds-cts-0" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "ds-cts-2" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "0" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650435" 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "ds-idrepo-0" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "0" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "3653.4666666666667" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "3881504768" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "ds-idrepo-0" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "0" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "0" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "599920" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650435" 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "11.742308388712422" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "ds-idrepo-0" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "0.00003717448126755506" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "ds-idrepo-0" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "ds-idrepo-2" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "0" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "ds-idrepo-0" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "ds-cts-2" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "0" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650435" 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "ds-cts-0" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "ds-cts-2" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "29" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "ds-idrepo-0" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "0" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "instance": "10.142.0.107:9100" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "instance": "10.142.0.43:9100" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "101039409.83333333" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "3329.5" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "3330.4" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650435" 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650435" 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:35 DEBUG "metric": { 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG "pod": "ds-idrepo-0" 23:00:35 DEBUG { 23:00:35 DEBUG }, 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "value": [ 23:00:35 DEBUG "data": { 23:00:35 DEBUG 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG 23:00:35 DEBUG "result": [ 23:00:35 DEBUG 23:00:35 DEBUG { 23:00:35 DEBUG [...] 23:00:35 DEBUG "metric": { 23:00:35 DEBUG 23:00:35 DEBUG "instance": "10.142.0.107:9100" 23:00:35 DEBUG 23:00:35 DEBUG }, 23:00:35 DEBUG 23:00:35 DEBUG "value": [ 23:00:35 DEBUG "pod": "ds-idrepo-2" 23:00:35 DEBUG 23:00:35 DEBUG }, 23:00:35 DEBUG 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG [...] 23:00:35 DEBUG "0" 23:00:35 DEBUG 23:00:35 DEBUG ] 23:00:35 DEBUG 23:00:35 DEBUG } 23:00:35 DEBUG 23:00:35 DEBUG ] 23:00:35 DEBUG "instance": "10.142.0.43:9100" 23:00:35 DEBUG } 23:00:35 DEBUG }, 23:00:35 DEBUG } 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "4260974941184" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650435" 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "instance": "10.142.0.107:9100" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "instance": "10.142.0.43:9100" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "101037061.03333333" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650435" 23:00:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "116.269" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:35 INFO [http_cmd]: http status code OK 23:00:35 DEBUG --- status code --- 23:00:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:00:35 DEBUG --- http response --- 23:00:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:00:35 DEBUG ----- output ----- 23:00:35 DEBUG { 23:00:35 DEBUG "status": "success", 23:00:35 DEBUG "data": { 23:00:35 DEBUG "resultType": "vector", 23:00:35 DEBUG "result": [ 23:00:35 DEBUG { 23:00:35 DEBUG "metric": { 23:00:35 DEBUG "instance": "10.142.0.107:9100" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG [...] 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG 23:00:35 DEBUG "instance": "10.142.0.43:9100" 23:00:35 DEBUG }, 23:00:35 DEBUG "value": [ 23:00:35 DEBUG 1713650435, 23:00:35 DEBUG "0.4188333333334034" 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG ] 23:00:35 DEBUG } 23:00:35 DEBUG } 23:00:36 INFO 23:00:36 INFO [loop_until]: kubectl --namespace=xlou top pods 23:00:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:00:36 INFO [loop_until]: OK (rc = 0) 23:00:36 DEBUG --- stdout --- 23:00:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22218m 8796Mi am-767f6b67b5-qcqz2 22424m 8921Mi am-767f6b67b5-vbltz 24821m 8990Mi ds-cts-0 20062m 12367Mi ds-cts-1 19135m 12480Mi ds-cts-2 20720m 12596Mi ds-idrepo-0 9465m 3914Mi ds-idrepo-1 9277m 3947Mi ds-idrepo-2 9638m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11556m 3612Mi 23:00:36 DEBUG --- stderr --- 23:00:36 DEBUG 23:00:37 INFO 23:00:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:00:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:00:37 INFO [loop_until]: OK (rc = 0) 23:00:37 DEBUG --- stdout --- 23:00:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6286Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9961m 33% 7686Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22089m 73% 11543Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3366Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20061m 67% 15609Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5221Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22606m 75% 11843Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19088m 63% 3140Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24940m 83% 11241Mi 10% gke-xlou-cdm-ds-0f601751-j488 10260m 34% 7292Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19686m 65% 17502Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10100m 33% 7940Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20711m 69% 15303Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11971m 75% 11463Mi 19% 23:00:37 DEBUG --- stderr --- 23:00:37 DEBUG 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650495" 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650495" 23:01:35 ERROR [http_cmd]: ERROR 23:01:35 ERROR --- status code --- 23:01:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:01:35 ERROR --- http response --- 23:01:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:01:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "pod": "ds-idrepo-0" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "ds-idrepo-2" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "0" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "pod": "ds-cts-0" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "ds-cts-2" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "0.5522999999999987" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "instance": "10.142.0.107:9100" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "instance": "10.142.0.43:9100" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "0.06900000000000735" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "data": { 23:01:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG }, 23:01:35 DEBUG "result": [ 23:01:35 DEBUG "value": [ 23:01:35 DEBUG { 23:01:35 DEBUG 23:01:35 DEBUG "metric": { 23:01:35 DEBUG 23:01:35 DEBUG "pod": "ds-cts-0" 23:01:35 DEBUG 23:01:35 DEBUG }, 23:01:35 DEBUG [...] 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:01:35 DEBUG [...] 23:01:35 DEBUG }, 23:01:35 DEBUG 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG 23:01:35 DEBUG "3323.3333333333335" 23:01:35 DEBUG "pod": "ds-cts-2" 23:01:35 DEBUG ] 23:01:35 DEBUG }, 23:01:35 DEBUG } 23:01:35 DEBUG "value": [ 23:01:35 DEBUG ] 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG } 23:01:35 DEBUG "0" 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650495" 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG { 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:01:35 DEBUG "3999.9666666666662" 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG ] 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG { 23:01:35 DEBUG } 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG ] 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG "data": { 23:01:35 DEBUG } 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:35 DEBUG } 23:01:35 DEBUG "result": [ 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "pod": "ds-idrepo-0" 23:01:35 DEBUG "data": { 23:01:35 DEBUG }, 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "value": [ 23:01:35 DEBUG "result": [ 23:01:35 DEBUG 23:01:35 DEBUG { 23:01:35 DEBUG 23:01:35 DEBUG "metric": { 23:01:35 DEBUG 23:01:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:01:35 DEBUG [...] 23:01:35 DEBUG }, 23:01:35 DEBUG 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:01:35 DEBUG 23:01:35 DEBUG }, 23:01:35 DEBUG [...] 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG 23:01:35 DEBUG "0" 23:01:35 DEBUG 23:01:35 DEBUG ] 23:01:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:01:35 DEBUG } 23:01:35 DEBUG }, 23:01:35 DEBUG ] 23:01:35 DEBUG "value": [ 23:01:35 DEBUG } 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG } 23:01:35 DEBUG "3884306432" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "pod": "ds-idrepo-0" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "0" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "0" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG "data": { 23:01:35 DEBUG { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "result": [ 23:01:35 DEBUG "pod": "ds-idrepo-0" 23:01:35 DEBUG { 23:01:35 DEBUG }, 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "value": [ 23:01:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:01:35 DEBUG 23:01:35 DEBUG }, 23:01:35 DEBUG 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "ds-idrepo-2" 23:01:35 DEBUG 23:01:35 DEBUG }, 23:01:35 DEBUG 23:01:35 DEBUG "value": [ 23:01:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG }, 23:01:35 DEBUG "0" 23:01:35 DEBUG "value": [ 23:01:35 DEBUG ] 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG } 23:01:35 DEBUG "599850" 23:01:35 DEBUG ] 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650495" 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "pod": "ds-idrepo-0" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "ds-cts-2" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "0" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "11.6994683949622" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "pod": "ds-idrepo-0" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "0.000008042997445870679" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "pod": "ds-idrepo-0" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "0" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650495" 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "pod": "ds-cts-0" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "ds-cts-2" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "28.7" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "instance": "10.142.0.107:9100" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "instance": "10.142.0.43:9100" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "101217441.13333333" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "3323.4666666666667" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "3325.4666666666667" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650495" 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "instance": "10.142.0.107:9100" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "instance": "10.142.0.43:9100" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "101213634.33333334" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650495" 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "pod": "ds-idrepo-0" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "ds-idrepo-2" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "0" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "instance": "10.142.0.107:9100" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "instance": "10.142.0.43:9100" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "4265939201024" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650495" 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650495" 23:01:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "120.2875" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:35 INFO [http_cmd]: http status code OK 23:01:35 DEBUG --- status code --- 23:01:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:01:35 DEBUG --- http response --- 23:01:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:01:35 DEBUG ----- output ----- 23:01:35 DEBUG { 23:01:35 DEBUG "status": "success", 23:01:35 DEBUG "data": { 23:01:35 DEBUG "resultType": "vector", 23:01:35 DEBUG "result": [ 23:01:35 DEBUG { 23:01:35 DEBUG "metric": { 23:01:35 DEBUG "instance": "10.142.0.107:9100" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG [...] 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG 23:01:35 DEBUG "instance": "10.142.0.43:9100" 23:01:35 DEBUG }, 23:01:35 DEBUG "value": [ 23:01:35 DEBUG 1713650495, 23:01:35 DEBUG "0.3635000000000351" 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG ] 23:01:35 DEBUG } 23:01:35 DEBUG } 23:01:36 INFO 23:01:36 INFO [loop_until]: kubectl --namespace=xlou top pods 23:01:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:01:36 INFO [loop_until]: OK (rc = 0) 23:01:36 DEBUG --- stdout --- 23:01:36 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22774m 8800Mi am-767f6b67b5-qcqz2 22911m 8919Mi am-767f6b67b5-vbltz 25027m 8991Mi ds-cts-0 20304m 12365Mi ds-cts-1 19852m 12491Mi ds-cts-2 20442m 12599Mi ds-idrepo-0 9679m 3915Mi ds-idrepo-1 9465m 3947Mi ds-idrepo-2 9557m 3869Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 4m 1238Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11687m 3612Mi 23:01:36 DEBUG --- stderr --- 23:01:36 DEBUG 23:01:37 INFO 23:01:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:01:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:01:37 INFO [loop_until]: OK (rc = 0) 23:01:37 DEBUG --- stdout --- 23:01:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6296Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10100m 33% 7687Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 21233m 71% 11537Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3371Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20142m 67% 15602Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5213Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22970m 76% 11843Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19679m 65% 3139Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 25159m 84% 11242Mi 10% gke-xlou-cdm-ds-0f601751-j488 10212m 34% 7289Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19557m 65% 17507Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10031m 33% 7927Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20347m 68% 15294Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11634m 73% 11466Mi 19% 23:01:37 DEBUG --- stderr --- 23:01:37 DEBUG 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650555" 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "ds-cts-0" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "ds-cts-2" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "0.5525666666666647" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "instance": "10.142.0.107:9100" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "instance": "10.142.0.43:9100" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "0.07666666666665758" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650555" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 ERROR [http_cmd]: ERROR 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 ERROR --- status code --- 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 ERROR --- http response --- 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "ds-idrepo-0" 23:02:35 DEBUG "pod": "ds-cts-0" 23:02:35 DEBUG }, 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "ds-idrepo-2" 23:02:35 DEBUG "pod": "ds-cts-2" 23:02:35 DEBUG }, 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "0" 23:02:35 DEBUG "0" 23:02:35 DEBUG ] 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "3358.3333333333335" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "3837.0666666666666" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650555" 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "overseer-0-967d957f7-zk66v" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "0" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "3884187648" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "0" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "0" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "ds-idrepo-0" 23:02:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:02:35 DEBUG }, 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG "value": [ 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "ds-idrepo-2" 23:02:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:02:35 DEBUG }, 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "0" 23:02:35 DEBUG "599819" 23:02:35 DEBUG ] 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "ds-idrepo-0" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "ds-cts-2" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "0" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "overseer-0-967d957f7-zk66v" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "0" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650555" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650555" 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "instance": "10.142.0.107:9100" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "instance": "10.142.0.43:9100" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "100964712.1" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "11.71685418588117" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "ds-cts-0" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "ds-cts-2" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "24.7" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "0" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:02:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:02:35 DEBUG }, 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:02:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:02:35 DEBUG }, 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "3332.9" 23:02:35 DEBUG "3359" 23:02:35 DEBUG ] 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650555" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650555" 23:02:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650555" 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG --- http response --- 23:02:35 INFO [http_cmd]: http status code OK 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:35 DEBUG --- status code --- 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:35 DEBUG { 23:02:35 DEBUG --- http response --- 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "pod": "ds-idrepo-0" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "pod": "ds-idrepo-2" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "0" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:35 DEBUG ----- output ----- 23:02:35 DEBUG { 23:02:35 DEBUG "status": "success", 23:02:35 DEBUG "data": { 23:02:35 DEBUG "resultType": "vector", 23:02:35 DEBUG "result": [ 23:02:35 DEBUG { 23:02:35 DEBUG "metric": { 23:02:35 DEBUG "instance": "10.142.0.107:9100" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG [...] 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG 23:02:35 DEBUG "instance": "10.142.0.43:9100" 23:02:35 DEBUG }, 23:02:35 DEBUG "value": [ 23:02:35 DEBUG 1713650555, 23:02:35 DEBUG "100963462.13333333" 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG ] 23:02:35 DEBUG } 23:02:35 DEBUG } 23:02:36 INFO [http_cmd]: http status code OK 23:02:36 DEBUG --- status code --- 23:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:36 DEBUG --- http response --- 23:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:36 DEBUG ----- output ----- 23:02:36 DEBUG { 23:02:36 DEBUG "status": "success", 23:02:36 DEBUG "data": { 23:02:36 DEBUG "resultType": "vector", 23:02:36 DEBUG "result": [ 23:02:36 DEBUG { 23:02:36 DEBUG "metric": { 23:02:36 DEBUG "instance": "10.142.0.107:9100" 23:02:36 DEBUG }, 23:02:36 DEBUG "value": [ 23:02:36 DEBUG 23:02:36 DEBUG 23:02:36 DEBUG 23:02:36 DEBUG [...] 23:02:36 DEBUG 23:02:36 DEBUG 23:02:36 DEBUG 23:02:36 DEBUG "instance": "10.142.0.43:9100" 23:02:36 DEBUG }, 23:02:36 DEBUG "value": [ 23:02:36 DEBUG 1713650555, 23:02:36 DEBUG "0.40209999999990204" 23:02:36 DEBUG ] 23:02:36 DEBUG } 23:02:36 DEBUG ] 23:02:36 DEBUG } 23:02:36 DEBUG } 23:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650556" 23:02:36 INFO [http_cmd]: http status code OK 23:02:36 DEBUG --- status code --- 23:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:36 DEBUG --- http response --- 23:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:02:36 DEBUG ----- output ----- 23:02:36 DEBUG { 23:02:36 DEBUG "status": "success", 23:02:36 DEBUG "data": { 23:02:36 DEBUG "resultType": "vector", 23:02:36 DEBUG "result": [ 23:02:36 DEBUG { 23:02:36 DEBUG "metric": { 23:02:36 DEBUG "instance": "10.142.0.107:9100" 23:02:36 DEBUG }, 23:02:36 DEBUG "value": [ 23:02:36 DEBUG 23:02:36 DEBUG 23:02:36 DEBUG 23:02:36 DEBUG [...] 23:02:36 DEBUG 23:02:36 DEBUG 23:02:36 DEBUG 23:02:36 DEBUG "instance": "10.142.0.43:9100" 23:02:36 DEBUG }, 23:02:36 DEBUG "value": [ 23:02:36 DEBUG 1713650556, 23:02:36 DEBUG "4271229283328" 23:02:36 DEBUG ] 23:02:36 DEBUG } 23:02:36 DEBUG ] 23:02:36 DEBUG } 23:02:36 DEBUG } 23:02:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650556" 23:02:36 INFO [http_cmd]: http status code OK 23:02:36 DEBUG --- status code --- 23:02:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:02:36 DEBUG --- http response --- 23:02:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:02:36 DEBUG ----- output ----- 23:02:36 DEBUG { 23:02:36 DEBUG "status": "success", 23:02:36 DEBUG "data": { 23:02:36 DEBUG "resultType": "vector", 23:02:36 DEBUG "result": [ 23:02:36 DEBUG { 23:02:36 DEBUG "metric": { 23:02:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:02:36 DEBUG }, 23:02:36 DEBUG "value": [ 23:02:36 DEBUG 23:02:36 DEBUG 23:02:36 DEBUG 23:02:36 DEBUG [...] 23:02:36 DEBUG 23:02:36 DEBUG 23:02:36 DEBUG 23:02:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:02:36 DEBUG }, 23:02:36 DEBUG "value": [ 23:02:36 DEBUG 1713650556, 23:02:36 DEBUG "125.46000000000001" 23:02:36 DEBUG ] 23:02:36 DEBUG } 23:02:36 DEBUG ] 23:02:36 DEBUG } 23:02:36 DEBUG } 23:02:37 INFO 23:02:37 INFO [loop_until]: kubectl --namespace=xlou top pods 23:02:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:02:37 INFO [loop_until]: OK (rc = 0) 23:02:37 DEBUG --- stdout --- 23:02:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22584m 8798Mi am-767f6b67b5-qcqz2 21957m 8921Mi am-767f6b67b5-vbltz 24633m 9001Mi ds-cts-0 20222m 12363Mi ds-cts-1 19340m 12483Mi ds-cts-2 20726m 12589Mi ds-idrepo-0 9826m 3914Mi ds-idrepo-1 9606m 3947Mi ds-idrepo-2 9205m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1179Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11725m 3616Mi 23:02:37 DEBUG --- stderr --- 23:02:37 DEBUG 23:02:37 INFO 23:02:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:02:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:02:37 INFO [loop_until]: OK (rc = 0) 23:02:37 DEBUG --- stdout --- 23:02:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6296Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10167m 34% 7690Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22167m 74% 11532Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3363Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20297m 67% 15611Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5193Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22492m 75% 11849Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19781m 66% 3135Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24975m 83% 11255Mi 10% gke-xlou-cdm-ds-0f601751-j488 10269m 34% 7285Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19659m 65% 17501Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10017m 33% 7924Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20612m 69% 15301Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11956m 75% 11465Mi 19% 23:02:37 DEBUG --- stderr --- 23:02:37 DEBUG 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650615" 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "ds-cts-0" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "ds-cts-2" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "0.5724000000000009" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "ds-idrepo-0" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "ds-idrepo-2" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "0" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "instance": "10.142.0.107:9100" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "instance": "10.142.0.43:9100" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "0.0776666666666794" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650615" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 ERROR [http_cmd]: ERROR 23:03:35 ERROR --- status code --- 23:03:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:03:35 ERROR --- http response --- 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG --- status code --- 23:03:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG { 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:03:35 DEBUG "data": { 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG }, 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "value": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "ds-cts-0" 23:03:35 DEBUG 23:03:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:03:35 DEBUG }, 23:03:35 DEBUG 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG [...] 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG }, 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG 23:03:35 DEBUG "pod": "ds-cts-2" 23:03:35 DEBUG "3832.8" 23:03:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:03:35 DEBUG }, 23:03:35 DEBUG ] 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG } 23:03:35 DEBUG "value": [ 23:03:35 DEBUG "0" 23:03:35 DEBUG } 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG ] 23:03:35 DEBUG "3319.0333333333333" 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "ds-idrepo-0" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "ds-idrepo-2" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "0" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650615" 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "0" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "3889692672" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "ds-idrepo-0" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "ds-idrepo-2" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "1036.7281074481075" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650615" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "600021" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "instance": "10.142.0.107:9100" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "instance": "10.142.0.43:9100" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "100722523.9" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "ds-idrepo-0" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "ds-idrepo-2" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "0" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG { 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "ds-idrepo-0" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "ds-cts-2" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "0" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:03:35 DEBUG } 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG } 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "ds-idrepo-0" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "ds-idrepo-2" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "2695.493079365079" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650615" 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG { 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "3321.2999999999997" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "ds-cts-0" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "ds-cts-2" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "29.466666666666665" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "11.738547622088683" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "ds-idrepo-0" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "ds-idrepo-2" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "6.649419584386028" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650615" 23:03:35 INFO [http_cmd]: http status code OK 23:03:35 DEBUG --- status code --- 23:03:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:35 DEBUG --- http response --- 23:03:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:35 DEBUG ----- output ----- 23:03:35 DEBUG { 23:03:35 DEBUG "status": "success", 23:03:35 DEBUG "data": { 23:03:35 DEBUG "resultType": "vector", 23:03:35 DEBUG "result": [ 23:03:35 DEBUG { 23:03:35 DEBUG "metric": { 23:03:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG [...] 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG 23:03:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:03:35 DEBUG }, 23:03:35 DEBUG "value": [ 23:03:35 DEBUG 1713650615, 23:03:35 DEBUG "3317.766666666667" 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG ] 23:03:35 DEBUG } 23:03:35 DEBUG } 23:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650616" 23:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650616" 23:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650616" 23:03:36 INFO [http_cmd]: http status code OK 23:03:36 DEBUG --- status code --- 23:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:36 DEBUG --- http response --- 23:03:36 INFO [http_cmd]: http status code OK 23:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:36 DEBUG --- status code --- 23:03:36 DEBUG ----- output ----- 23:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:36 DEBUG { 23:03:36 DEBUG --- http response --- 23:03:36 DEBUG "status": "success", 23:03:36 DEBUG "data": { 23:03:36 DEBUG "resultType": "vector", 23:03:36 DEBUG "result": [ 23:03:36 DEBUG { 23:03:36 DEBUG "metric": { 23:03:36 DEBUG "pod": "ds-idrepo-0" 23:03:36 DEBUG }, 23:03:36 DEBUG "value": [ 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG [...] 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:36 DEBUG "pod": "ds-idrepo-2" 23:03:36 DEBUG ----- output ----- 23:03:36 DEBUG }, 23:03:36 DEBUG { 23:03:36 DEBUG "value": [ 23:03:36 DEBUG "status": "success", 23:03:36 DEBUG 1713650616, 23:03:36 DEBUG "data": { 23:03:36 DEBUG "0" 23:03:36 DEBUG "resultType": "vector", 23:03:36 DEBUG ] 23:03:36 DEBUG "result": [ 23:03:36 DEBUG } 23:03:36 DEBUG { 23:03:36 DEBUG ] 23:03:36 DEBUG "metric": { 23:03:36 DEBUG } 23:03:36 DEBUG "instance": "10.142.0.107:9100" 23:03:36 DEBUG } 23:03:36 DEBUG }, 23:03:36 DEBUG "value": [ 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG [...] 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG "instance": "10.142.0.43:9100" 23:03:36 DEBUG }, 23:03:36 DEBUG "value": [ 23:03:36 DEBUG 1713650616, 23:03:36 DEBUG "100721334.23333332" 23:03:36 DEBUG ] 23:03:36 DEBUG } 23:03:36 DEBUG ] 23:03:36 DEBUG } 23:03:36 DEBUG } 23:03:36 INFO [http_cmd]: http status code OK 23:03:36 DEBUG --- status code --- 23:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:36 DEBUG --- http response --- 23:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:36 DEBUG ----- output ----- 23:03:36 DEBUG { 23:03:36 DEBUG "status": "success", 23:03:36 DEBUG "data": { 23:03:36 DEBUG "resultType": "vector", 23:03:36 DEBUG "result": [ 23:03:36 DEBUG { 23:03:36 DEBUG "metric": { 23:03:36 DEBUG "instance": "10.142.0.107:9100" 23:03:36 DEBUG }, 23:03:36 DEBUG "value": [ 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG [...] 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG "instance": "10.142.0.43:9100" 23:03:36 DEBUG }, 23:03:36 DEBUG "value": [ 23:03:36 DEBUG 1713650616, 23:03:36 DEBUG "0.400933333333334" 23:03:36 DEBUG ] 23:03:36 DEBUG } 23:03:36 DEBUG ] 23:03:36 DEBUG } 23:03:36 DEBUG } 23:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650616" 23:03:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650616" 23:03:36 INFO [http_cmd]: http status code OK 23:03:36 DEBUG --- status code --- 23:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:36 DEBUG --- http response --- 23:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:03:36 DEBUG ----- output ----- 23:03:36 DEBUG { 23:03:36 DEBUG "status": "success", 23:03:36 DEBUG "data": { 23:03:36 DEBUG "resultType": "vector", 23:03:36 DEBUG "result": [ 23:03:36 DEBUG { 23:03:36 DEBUG "metric": { 23:03:36 DEBUG "instance": "10.142.0.107:9100" 23:03:36 DEBUG }, 23:03:36 DEBUG "value": [ 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG [...] 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG "instance": "10.142.0.43:9100" 23:03:36 DEBUG }, 23:03:36 DEBUG "value": [ 23:03:36 DEBUG 1713650616, 23:03:36 DEBUG "4276275262464" 23:03:36 DEBUG ] 23:03:36 DEBUG } 23:03:36 DEBUG ] 23:03:36 DEBUG } 23:03:36 DEBUG } 23:03:36 INFO [http_cmd]: http status code OK 23:03:36 DEBUG --- status code --- 23:03:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:03:36 DEBUG --- http response --- 23:03:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:03:36 DEBUG ----- output ----- 23:03:36 DEBUG { 23:03:36 DEBUG "status": "success", 23:03:36 DEBUG "data": { 23:03:36 DEBUG "resultType": "vector", 23:03:36 DEBUG "result": [ 23:03:36 DEBUG { 23:03:36 DEBUG "metric": { 23:03:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:03:36 DEBUG }, 23:03:36 DEBUG "value": [ 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG [...] 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG 23:03:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:03:36 DEBUG }, 23:03:36 DEBUG "value": [ 23:03:36 DEBUG 1713650616, 23:03:36 DEBUG "129.651" 23:03:36 DEBUG ] 23:03:36 DEBUG } 23:03:36 DEBUG ] 23:03:36 DEBUG } 23:03:36 DEBUG } 23:03:37 INFO 23:03:37 INFO [loop_until]: kubectl --namespace=xlou top pods 23:03:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:03:37 INFO [loop_until]: OK (rc = 0) 23:03:37 DEBUG --- stdout --- 23:03:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 22893m 8798Mi am-767f6b67b5-qcqz2 22538m 8933Mi am-767f6b67b5-vbltz 24591m 8994Mi ds-cts-0 20932m 12360Mi ds-cts-1 19708m 12489Mi ds-cts-2 20404m 12589Mi ds-idrepo-0 9718m 3915Mi ds-idrepo-1 9401m 3947Mi ds-idrepo-2 9578m 3868Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 11679m 3620Mi 23:03:37 DEBUG --- stderr --- 23:03:37 DEBUG 23:03:37 INFO 23:03:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:03:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:03:37 INFO [loop_until]: OK (rc = 0) 23:03:37 DEBUG --- stdout --- 23:03:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6295Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10102m 33% 7682Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 22180m 74% 11549Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3371Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 20577m 68% 15599Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5194Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 22038m 73% 11863Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19726m 66% 3140Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24988m 83% 11247Mi 10% gke-xlou-cdm-ds-0f601751-j488 10311m 34% 7287Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19810m 66% 17498Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 9968m 33% 7926Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20505m 68% 15301Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 11967m 75% 11464Mi 19% 23:03:37 DEBUG --- stderr --- 23:03:37 DEBUG 23:03:50 WARNING File /tmp/lodemon_data-20240420_223134/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [20/Apr/2024 23:03:50] "GET /monitoring/average?start_time=20240420_213436&stop_time=20240420_220149 HTTP/1.1" 200 - 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650675" 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "pod": "ds-cts-0" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "pod": "ds-cts-2" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG "26.3664" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650675" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650675" 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG "3828.2666666666664" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650675" 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "pod": "ds-idrepo-0" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "pod": "ds-idrepo-2" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG "0" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650675" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650675" 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG { 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:35 DEBUG "result": [ 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:04:35 DEBUG "data": { 23:04:35 DEBUG }, 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "value": [ 23:04:35 DEBUG "result": [ 23:04:35 DEBUG 23:04:35 DEBUG { 23:04:35 DEBUG 23:04:35 DEBUG "metric": { 23:04:35 DEBUG 23:04:35 DEBUG "instance": "10.142.0.107:9100" 23:04:35 DEBUG [...] 23:04:35 DEBUG }, 23:04:35 DEBUG 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 ERROR [http_cmd]: ERROR 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 ERROR --- status code --- 23:04:35 DEBUG 23:04:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:04:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:04:35 DEBUG 23:04:35 DEBUG }, 23:04:35 ERROR --- http response --- 23:04:35 DEBUG [...] 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG 23:04:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:04:35 DEBUG "240.16666666666666" 23:04:35 DEBUG 23:04:35 DEBUG ] 23:04:35 DEBUG "instance": "10.142.0.43:9100" 23:04:35 DEBUG } 23:04:35 DEBUG }, 23:04:35 DEBUG ] 23:04:35 DEBUG "value": [ 23:04:35 DEBUG } 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG } 23:04:35 DEBUG "0.09500000000001023" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650675" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650675" 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "pod": "ds-cts-0" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "pod": "ds-cts-2" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG "0" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "pod": "ds-idrepo-1" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG "0" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650675" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650675" 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG "0" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650675" 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG "3232681984" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650675" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650675" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650675" 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG { 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "pod": "ds-idrepo-0" 23:04:35 DEBUG }, 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:04:35 DEBUG "value": [ 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG 23:04:35 DEBUG { 23:04:35 DEBUG 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG 23:04:35 DEBUG "data": { 23:04:35 DEBUG [...] 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG 23:04:35 DEBUG "result": [ 23:04:35 DEBUG 23:04:35 DEBUG { 23:04:35 DEBUG 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "pod": "ds-idrepo-2" 23:04:35 DEBUG "pod": "ds-idrepo-1" 23:04:35 DEBUG }, 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG 23:04:35 DEBUG "0" 23:04:35 DEBUG 23:04:35 DEBUG ] 23:04:35 DEBUG 23:04:35 DEBUG } 23:04:35 DEBUG [...] 23:04:35 DEBUG ] 23:04:35 DEBUG 23:04:35 DEBUG } 23:04:35 DEBUG 23:04:35 DEBUG } 23:04:35 DEBUG 23:04:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG "0" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "pod": "ds-idrepo-1" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG "0" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650675" 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650675" 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG "472890.5" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650675" 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "instance": "10.142.0.107:9100" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "instance": "10.142.0.43:9100" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG "24335678.233333334" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:35 INFO [http_cmd]: http status code OK 23:04:35 DEBUG --- status code --- 23:04:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:35 DEBUG --- http response --- 23:04:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:04:35 DEBUG ----- output ----- 23:04:35 DEBUG { 23:04:35 DEBUG "status": "success", 23:04:35 DEBUG "data": { 23:04:35 DEBUG "resultType": "vector", 23:04:35 DEBUG "result": [ 23:04:35 DEBUG { 23:04:35 DEBUG "metric": { 23:04:35 DEBUG "pod": "ds-idrepo-0" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG [...] 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG 23:04:35 DEBUG "pod": "ds-cts-2" 23:04:35 DEBUG }, 23:04:35 DEBUG "value": [ 23:04:35 DEBUG 1713650675, 23:04:35 DEBUG "0" 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG ] 23:04:35 DEBUG } 23:04:35 DEBUG } 23:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650676" 23:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650676" 23:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650676" 23:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650676" 23:04:36 INFO [http_cmd]: http status code OK 23:04:36 INFO [http_cmd]: http status code OK 23:04:36 INFO [http_cmd]: http status code OK 23:04:36 DEBUG --- status code --- 23:04:36 DEBUG --- status code --- 23:04:36 DEBUG --- status code --- 23:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:36 DEBUG --- http response --- 23:04:36 DEBUG --- http response --- 23:04:36 DEBUG --- http response --- 23:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:36 DEBUG ----- output ----- 23:04:36 DEBUG ----- output ----- 23:04:36 DEBUG ----- output ----- 23:04:36 DEBUG { 23:04:36 DEBUG { 23:04:36 DEBUG { 23:04:36 DEBUG "status": "success", 23:04:36 DEBUG "status": "success", 23:04:36 DEBUG "status": "success", 23:04:36 DEBUG "data": { 23:04:36 DEBUG "data": { 23:04:36 DEBUG "data": { 23:04:36 DEBUG "resultType": "vector", 23:04:36 DEBUG "resultType": "vector", 23:04:36 DEBUG "resultType": "vector", 23:04:36 DEBUG "result": [ 23:04:36 DEBUG "result": [ 23:04:36 DEBUG "result": [ 23:04:36 DEBUG { 23:04:36 DEBUG { 23:04:36 DEBUG { 23:04:36 DEBUG "metric": { 23:04:36 DEBUG "metric": { 23:04:36 DEBUG "metric": { 23:04:36 DEBUG "pod": "ds-idrepo-1" 23:04:36 DEBUG "pod": "ds-cts-0" 23:04:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:04:36 DEBUG }, 23:04:36 DEBUG }, 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG "value": [ 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG [...] 23:04:36 DEBUG [...] 23:04:36 DEBUG [...] 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 INFO [http_cmd]: http status code OK 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG --- status code --- 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:36 DEBUG 23:04:36 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:04:36 DEBUG "pod": "ds-cts-2" 23:04:36 DEBUG --- http response --- 23:04:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:04:36 DEBUG }, 23:04:36 DEBUG }, 23:04:36 DEBUG }, 23:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:36 DEBUG "value": [ 23:04:36 DEBUG "value": [ 23:04:36 DEBUG "value": [ 23:04:36 DEBUG ----- output ----- 23:04:36 DEBUG 1713650676, 23:04:36 DEBUG 1713650676, 23:04:36 DEBUG 1713650676, 23:04:36 DEBUG { 23:04:36 DEBUG "0.000005548711423941171" 23:04:36 DEBUG "29.4" 23:04:36 DEBUG "239.73333333333332" 23:04:36 DEBUG "status": "success", 23:04:36 DEBUG ] 23:04:36 DEBUG ] 23:04:36 DEBUG ] 23:04:36 DEBUG "data": { 23:04:36 DEBUG } 23:04:36 DEBUG } 23:04:36 DEBUG } 23:04:36 DEBUG "resultType": "vector", 23:04:36 DEBUG ] 23:04:36 DEBUG ] 23:04:36 DEBUG ] 23:04:36 DEBUG "result": [ 23:04:36 DEBUG } 23:04:36 DEBUG } 23:04:36 DEBUG } 23:04:36 DEBUG { 23:04:36 DEBUG } 23:04:36 DEBUG } 23:04:36 DEBUG } 23:04:36 DEBUG "metric": { 23:04:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG [...] 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 1713650676, 23:04:36 DEBUG "1.3847837104579834" 23:04:36 DEBUG ] 23:04:36 DEBUG } 23:04:36 DEBUG ] 23:04:36 DEBUG } 23:04:36 DEBUG } 23:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650676" 23:04:36 INFO [http_cmd]: http status code OK 23:04:36 DEBUG --- status code --- 23:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:36 DEBUG --- http response --- 23:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:36 DEBUG ----- output ----- 23:04:36 DEBUG { 23:04:36 DEBUG "status": "success", 23:04:36 DEBUG "data": { 23:04:36 DEBUG "resultType": "vector", 23:04:36 DEBUG "result": [ 23:04:36 DEBUG { 23:04:36 DEBUG "metric": { 23:04:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG [...] 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 1713650676, 23:04:36 DEBUG "240.96666666666667" 23:04:36 DEBUG ] 23:04:36 DEBUG } 23:04:36 DEBUG ] 23:04:36 DEBUG } 23:04:36 DEBUG } 23:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650676" 23:04:36 INFO [http_cmd]: http status code OK 23:04:36 DEBUG --- status code --- 23:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:36 DEBUG --- http response --- 23:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:36 DEBUG ----- output ----- 23:04:36 DEBUG { 23:04:36 DEBUG "status": "success", 23:04:36 DEBUG "data": { 23:04:36 DEBUG "resultType": "vector", 23:04:36 DEBUG "result": [ 23:04:36 DEBUG { 23:04:36 DEBUG "metric": { 23:04:36 DEBUG "pod": "ds-idrepo-0" 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG [...] 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG "pod": "ds-idrepo-2" 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 1713650676, 23:04:36 DEBUG "0" 23:04:36 DEBUG ] 23:04:36 DEBUG } 23:04:36 DEBUG ] 23:04:36 DEBUG } 23:04:36 DEBUG } 23:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650676" 23:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650676" 23:04:36 INFO [http_cmd]: http status code OK 23:04:36 DEBUG --- status code --- 23:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:36 DEBUG --- http response --- 23:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:36 DEBUG ----- output ----- 23:04:36 DEBUG { 23:04:36 DEBUG "status": "success", 23:04:36 DEBUG "data": { 23:04:36 DEBUG "resultType": "vector", 23:04:36 DEBUG "result": [ 23:04:36 DEBUG { 23:04:36 DEBUG "metric": { 23:04:36 DEBUG "instance": "10.142.0.107:9100" 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG [...] 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG "instance": "10.142.0.43:9100" 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 1713650676, 23:04:36 DEBUG "24336993.166666664" 23:04:36 DEBUG ] 23:04:36 DEBUG } 23:04:36 DEBUG ] 23:04:36 DEBUG } 23:04:36 DEBUG } 23:04:36 INFO [http_cmd]: http status code OK 23:04:36 DEBUG --- status code --- 23:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:36 DEBUG --- http response --- 23:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:36 DEBUG ----- output ----- 23:04:36 DEBUG { 23:04:36 DEBUG "status": "success", 23:04:36 DEBUG "data": { 23:04:36 DEBUG "resultType": "vector", 23:04:36 DEBUG "result": [ 23:04:36 DEBUG { 23:04:36 DEBUG "metric": { 23:04:36 DEBUG "instance": "10.142.0.107:9100" 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG [...] 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG "instance": "10.142.0.43:9100" 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 1713650676, 23:04:36 DEBUG "0.4983999999999772" 23:04:36 DEBUG ] 23:04:36 DEBUG } 23:04:36 DEBUG ] 23:04:36 DEBUG } 23:04:36 DEBUG } 23:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650676" 23:04:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650676" 23:04:36 INFO [http_cmd]: http status code OK 23:04:36 DEBUG --- status code --- 23:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:36 DEBUG --- http response --- 23:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:04:36 DEBUG ----- output ----- 23:04:36 DEBUG { 23:04:36 DEBUG "status": "success", 23:04:36 DEBUG "data": { 23:04:36 DEBUG "resultType": "vector", 23:04:36 DEBUG "result": [ 23:04:36 DEBUG { 23:04:36 DEBUG "metric": { 23:04:36 DEBUG "instance": "10.142.0.107:9100" 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG [...] 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG "instance": "10.142.0.43:9100" 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 1713650676, 23:04:36 DEBUG "4280794455040" 23:04:36 DEBUG ] 23:04:36 DEBUG } 23:04:36 DEBUG ] 23:04:36 DEBUG } 23:04:36 DEBUG } 23:04:36 INFO [http_cmd]: http status code OK 23:04:36 DEBUG --- status code --- 23:04:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:04:36 DEBUG --- http response --- 23:04:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:04:36 DEBUG ----- output ----- 23:04:36 DEBUG { 23:04:36 DEBUG "status": "success", 23:04:36 DEBUG "data": { 23:04:36 DEBUG "resultType": "vector", 23:04:36 DEBUG "result": [ 23:04:36 DEBUG { 23:04:36 DEBUG "metric": { 23:04:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG [...] 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG 23:04:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:04:36 DEBUG }, 23:04:36 DEBUG "value": [ 23:04:36 DEBUG 1713650676, 23:04:36 DEBUG "131.79149999999998" 23:04:36 DEBUG ] 23:04:36 DEBUG } 23:04:36 DEBUG ] 23:04:36 DEBUG } 23:04:36 DEBUG } 23:04:37 INFO 23:04:37 INFO [loop_until]: kubectl --namespace=xlou top pods 23:04:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:04:37 INFO [loop_until]: OK (rc = 0) 23:04:37 DEBUG --- stdout --- 23:04:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 3585m 8797Mi am-767f6b67b5-qcqz2 4629m 8922Mi am-767f6b67b5-vbltz 5933m 8986Mi ds-cts-0 18977m 12354Mi ds-cts-1 18421m 12474Mi ds-cts-2 19033m 12579Mi ds-idrepo-0 941m 3914Mi ds-idrepo-1 914m 3946Mi ds-idrepo-2 1271m 3869Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1178Mi idm-b85f9d978-wnhxb 4m 1239Mi lodemon-5497cb87d7-mlld4 3m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 3024m 3158Mi 23:04:37 DEBUG --- stderr --- 23:04:37 DEBUG 23:04:37 INFO 23:04:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:04:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:04:37 INFO [loop_until]: OK (rc = 0) 23:04:37 DEBUG --- stdout --- 23:04:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6302Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1972m 6% 7684Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6163m 20% 11554Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3368Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18904m 63% 15606Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5186Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 4934m 16% 11863Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 1198m 4% 3102Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 6826m 22% 11242Mi 10% gke-xlou-cdm-ds-0f601751-j488 2082m 6% 7289Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18224m 61% 17480Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 2006m 6% 7930Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19563m 65% 15298Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 3722m 23% 11023Mi 18% 23:04:37 DEBUG --- stderr --- 23:04:37 DEBUG 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650735" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650735" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650735" 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 1713650735, 23:05:35 DEBUG "3735.3333333333335" 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "pod": "ds-cts-0" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "pod": "ds-cts-2" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 1713650735, 23:05:35 DEBUG "0.5506666666666661" 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "instance": "10.142.0.107:9100" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "instance": "10.142.0.43:9100" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 1713650735, 23:05:35 DEBUG "0.08699999999999286" 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650735" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650735" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650735" 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "pod": "ds-cts-0" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "pod": "ds-cts-2" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 1713650735, 23:05:35 DEBUG "0" 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650735" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650735" 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "pod": "ds-idrepo-0" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "pod": "ds-idrepo-2" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 1713650735, 23:05:35 DEBUG "0" 23:05:35 DEBUG ] 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG } 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG ] 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG } 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG } 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 1713650735, 23:05:35 DEBUG "0" 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 ERROR [http_cmd]: ERROR 23:05:35 ERROR --- status code --- 23:05:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:05:35 ERROR --- http response --- 23:05:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:05:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 1713650735, 23:05:35 DEBUG "3461.0333333333333" 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650735" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650735" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650735" 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG ----- output ----- 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650735" 23:05:35 DEBUG { 23:05:35 DEBUG { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "data": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "result": [ 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:05:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:05:35 DEBUG }, 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:05:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:05:35 DEBUG }, 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 1713650735, 23:05:35 DEBUG 1713650735, 23:05:35 DEBUG "0" 23:05:35 DEBUG "580.1494281363974" 23:05:35 DEBUG ] 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 1713650735, 23:05:35 DEBUG "4060393472" 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650735" 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650735" 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "pod": "ds-idrepo-0" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "pod": "ds-idrepo-2" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 1713650735, 23:05:35 DEBUG "0" 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650735" 23:05:35 INFO [http_cmd]: http status code OK 23:05:35 DEBUG --- status code --- 23:05:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:35 DEBUG --- http response --- 23:05:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:05:35 DEBUG ----- output ----- 23:05:35 DEBUG { 23:05:35 DEBUG "status": "success", 23:05:35 DEBUG "data": { 23:05:35 DEBUG "resultType": "vector", 23:05:35 DEBUG "result": [ 23:05:35 DEBUG { 23:05:35 DEBUG "metric": { 23:05:35 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG [...] 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG 23:05:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:05:35 DEBUG }, 23:05:35 DEBUG "value": [ 23:05:35 DEBUG 1713650735, 23:05:35 DEBUG "10297.652349421054" 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG ] 23:05:35 DEBUG } 23:05:35 DEBUG } 23:05:36 INFO [http_cmd]: http status code OK 23:05:36 DEBUG --- status code --- 23:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:36 DEBUG --- http response --- 23:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:05:36 DEBUG ----- output ----- 23:05:36 DEBUG { 23:05:36 DEBUG "status": "success", 23:05:36 DEBUG "data": { 23:05:36 DEBUG "resultType": "vector", 23:05:36 DEBUG "result": [ 23:05:36 DEBUG { 23:05:36 DEBUG "metric": { 23:05:36 DEBUG "pod": "ds-idrepo-0" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG [...] 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG "pod": "ds-cts-2" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 1713650735, 23:05:36 DEBUG "0" 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG } 23:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650736" 23:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650736" 23:05:36 INFO [http_cmd]: http status code OK 23:05:36 DEBUG --- status code --- 23:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:36 DEBUG --- http response --- 23:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:36 DEBUG ----- output ----- 23:05:36 DEBUG { 23:05:36 DEBUG "status": "success", 23:05:36 DEBUG "data": { 23:05:36 DEBUG "resultType": "vector", 23:05:36 DEBUG "result": [ 23:05:36 DEBUG { 23:05:36 DEBUG "metric": { 23:05:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG [...] 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 1713650736, 23:05:36 DEBUG "412684.5" 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG } 23:05:36 INFO [http_cmd]: http status code OK 23:05:36 DEBUG --- status code --- 23:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:36 DEBUG --- http response --- 23:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:36 DEBUG ----- output ----- 23:05:36 DEBUG { 23:05:36 DEBUG "status": "success", 23:05:36 DEBUG "data": { 23:05:36 DEBUG "resultType": "vector", 23:05:36 DEBUG "result": [ 23:05:36 DEBUG { 23:05:36 DEBUG "metric": { 23:05:36 DEBUG "instance": "10.142.0.107:9100" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG [...] 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG "instance": "10.142.0.43:9100" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 1713650736, 23:05:36 DEBUG "106790733.46666667" 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG } 23:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650736" 23:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650736" 23:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650736" 23:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650736" 23:05:36 INFO [http_cmd]: http status code OK 23:05:36 DEBUG --- status code --- 23:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:36 DEBUG --- http response --- 23:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:05:36 DEBUG ----- output ----- 23:05:36 DEBUG { 23:05:36 DEBUG "status": "success", 23:05:36 DEBUG "data": { 23:05:36 DEBUG "resultType": "vector", 23:05:36 DEBUG "result": [ 23:05:36 DEBUG { 23:05:36 DEBUG "metric": { 23:05:36 DEBUG "pod": "ds-cts-0" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG [...] 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG "pod": "ds-cts-2" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 1713650736, 23:05:36 DEBUG "29.03333333333333" 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG } 23:05:36 INFO [http_cmd]: http status code OK 23:05:36 INFO [http_cmd]: http status code OK 23:05:36 DEBUG --- status code --- 23:05:36 DEBUG --- status code --- 23:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:36 DEBUG --- http response --- 23:05:36 DEBUG --- http response --- 23:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:05:36 DEBUG ----- output ----- 23:05:36 DEBUG { 23:05:36 DEBUG "status": "success", 23:05:36 DEBUG "data": { 23:05:36 DEBUG "resultType": "vector", 23:05:36 DEBUG "result": [ 23:05:36 DEBUG { 23:05:36 DEBUG "metric": { 23:05:36 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG [...] 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 1713650736, 23:05:36 DEBUG "0" 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG } 23:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:36 DEBUG ----- output ----- 23:05:36 DEBUG { 23:05:36 DEBUG "status": "success", 23:05:36 DEBUG "data": { 23:05:36 DEBUG "resultType": "vector", 23:05:36 DEBUG "result": [ 23:05:36 DEBUG { 23:05:36 DEBUG "metric": { 23:05:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG [...] 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 1713650736, 23:05:36 DEBUG "3463.0666666666666" 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG } 23:05:36 INFO [http_cmd]: http status code OK 23:05:36 DEBUG --- status code --- 23:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:36 DEBUG --- http response --- 23:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:36 DEBUG ----- output ----- 23:05:36 DEBUG { 23:05:36 DEBUG "status": "success", 23:05:36 DEBUG "data": { 23:05:36 DEBUG "resultType": "vector", 23:05:36 DEBUG "result": [ 23:05:36 DEBUG { 23:05:36 DEBUG "metric": { 23:05:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG [...] 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 1713650736, 23:05:36 DEBUG "11.545206586481031" 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG } 23:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650736" 23:05:36 INFO [http_cmd]: http status code OK 23:05:36 DEBUG --- status code --- 23:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:36 DEBUG --- http response --- 23:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:36 DEBUG ----- output ----- 23:05:36 DEBUG { 23:05:36 DEBUG "status": "success", 23:05:36 DEBUG "data": { 23:05:36 DEBUG "resultType": "vector", 23:05:36 DEBUG "result": [ 23:05:36 DEBUG { 23:05:36 DEBUG "metric": { 23:05:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG [...] 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 1713650736, 23:05:36 DEBUG "3461.5666666666666" 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG } 23:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650736" 23:05:36 INFO [http_cmd]: http status code OK 23:05:36 DEBUG --- status code --- 23:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:36 DEBUG --- http response --- 23:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:36 DEBUG ----- output ----- 23:05:36 DEBUG { 23:05:36 DEBUG "status": "success", 23:05:36 DEBUG "data": { 23:05:36 DEBUG "resultType": "vector", 23:05:36 DEBUG "result": [ 23:05:36 DEBUG { 23:05:36 DEBUG "metric": { 23:05:36 DEBUG "pod": "ds-idrepo-0" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG [...] 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG "pod": "ds-idrepo-2" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 1713650736, 23:05:36 DEBUG "0" 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG } 23:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650736" 23:05:36 INFO [http_cmd]: http status code OK 23:05:36 DEBUG --- status code --- 23:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:36 DEBUG --- http response --- 23:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:36 DEBUG ----- output ----- 23:05:36 DEBUG { 23:05:36 DEBUG "status": "success", 23:05:36 DEBUG "data": { 23:05:36 DEBUG "resultType": "vector", 23:05:36 DEBUG "result": [ 23:05:36 DEBUG { 23:05:36 DEBUG "metric": { 23:05:36 DEBUG "instance": "10.142.0.107:9100" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG [...] 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG "instance": "10.142.0.43:9100" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 1713650736, 23:05:36 DEBUG "106787358.7" 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG } 23:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650736" 23:05:36 INFO [http_cmd]: http status code OK 23:05:36 DEBUG --- status code --- 23:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:36 DEBUG --- http response --- 23:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:36 DEBUG ----- output ----- 23:05:36 DEBUG { 23:05:36 DEBUG "status": "success", 23:05:36 DEBUG "data": { 23:05:36 DEBUG "resultType": "vector", 23:05:36 DEBUG "result": [ 23:05:36 DEBUG { 23:05:36 DEBUG "metric": { 23:05:36 DEBUG "instance": "10.142.0.107:9100" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG [...] 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG "instance": "10.142.0.43:9100" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 1713650736, 23:05:36 DEBUG "0.42059999999998465" 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG } 23:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650736" 23:05:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650736" 23:05:36 INFO [http_cmd]: http status code OK 23:05:36 DEBUG --- status code --- 23:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:36 DEBUG --- http response --- 23:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:05:36 DEBUG ----- output ----- 23:05:36 DEBUG { 23:05:36 DEBUG "status": "success", 23:05:36 DEBUG "data": { 23:05:36 DEBUG "resultType": "vector", 23:05:36 DEBUG "result": [ 23:05:36 DEBUG { 23:05:36 DEBUG "metric": { 23:05:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG [...] 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 1713650736, 23:05:36 DEBUG "133.5585" 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG } 23:05:36 INFO [http_cmd]: http status code OK 23:05:36 DEBUG --- status code --- 23:05:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:05:36 DEBUG --- http response --- 23:05:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:05:36 DEBUG ----- output ----- 23:05:36 DEBUG { 23:05:36 DEBUG "status": "success", 23:05:36 DEBUG "data": { 23:05:36 DEBUG "resultType": "vector", 23:05:36 DEBUG "result": [ 23:05:36 DEBUG { 23:05:36 DEBUG "metric": { 23:05:36 DEBUG "instance": "10.142.0.107:9100" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG [...] 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG 23:05:36 DEBUG "instance": "10.142.0.43:9100" 23:05:36 DEBUG }, 23:05:36 DEBUG "value": [ 23:05:36 DEBUG 1713650736, 23:05:36 DEBUG "4285475779584" 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG ] 23:05:36 DEBUG } 23:05:36 DEBUG } 23:05:37 INFO 23:05:37 INFO [loop_until]: kubectl --namespace=xlou top pods 23:05:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:05:37 INFO [loop_until]: OK (rc = 0) 23:05:37 DEBUG --- stdout --- 23:05:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 24694m 8801Mi am-767f6b67b5-qcqz2 24288m 8928Mi am-767f6b67b5-vbltz 24858m 9032Mi ds-cts-0 20002m 12362Mi ds-cts-1 19423m 12481Mi ds-cts-2 21079m 12590Mi ds-idrepo-0 10876m 3914Mi ds-idrepo-1 10204m 3947Mi ds-idrepo-2 10472m 3869Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 13278m 4678Mi 23:05:37 DEBUG --- stderr --- 23:05:37 DEBUG 23:05:37 INFO 23:05:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:05:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:05:37 INFO [loop_until]: OK (rc = 0) 23:05:37 DEBUG --- stdout --- 23:05:37 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6282Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10963m 36% 7690Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 24701m 82% 11546Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3370Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19886m 66% 15598Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5198Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 24729m 82% 11850Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 23663m 79% 3315Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 24806m 83% 11340Mi 10% gke-xlou-cdm-ds-0f601751-j488 10925m 36% 7290Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19187m 64% 17512Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 10717m 35% 7924Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20505m 68% 15295Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 13494m 84% 12525Mi 21% 23:05:37 DEBUG --- stderr --- 23:05:37 DEBUG 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650795" 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG { 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "data": { 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "result": [ 23:06:35 DEBUG { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "instance": "10.142.0.107:9100" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "instance": "10.142.0.43:9100" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 1713650795, 23:06:35 DEBUG "0.09333333333332991" 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650795" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650795" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650795" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650795" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650795" 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG { 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "data": { 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "result": [ 23:06:35 DEBUG { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 1713650795, 23:06:35 DEBUG "4181.666666666667" 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG { 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "data": { 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "result": [ 23:06:35 DEBUG { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "pod": "ds-cts-0" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "pod": "ds-cts-2" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 1713650795, 23:06:35 DEBUG "0.37299999999999994" 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG { 23:06:35 DEBUG { 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "data": { 23:06:35 DEBUG "data": { 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "result": [ 23:06:35 DEBUG "result": [ 23:06:35 DEBUG { 23:06:35 DEBUG { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "pod": "ds-cts-0" 23:06:35 DEBUG "pod": "ds-idrepo-0" 23:06:35 DEBUG }, 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "pod": "ds-cts-2" 23:06:35 DEBUG "pod": "ds-idrepo-2" 23:06:35 DEBUG }, 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 1713650795, 23:06:35 DEBUG 1713650795, 23:06:35 DEBUG "0" 23:06:35 DEBUG "0" 23:06:35 DEBUG ] 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG ] 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 ERROR [http_cmd]: ERROR 23:06:35 ERROR --- status code --- 23:06:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:06:35 ERROR --- http response --- 23:06:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:06:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650795" 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650795" 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG { 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "data": { 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "result": [ 23:06:35 DEBUG { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 1713650795, 23:06:35 DEBUG "2423.9" 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 INFO [http_cmd]: http status code OK 23:06:35 DEBUG --- status code --- 23:06:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:35 DEBUG --- http response --- 23:06:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:06:35 DEBUG ----- output ----- 23:06:35 DEBUG { 23:06:35 DEBUG "status": "success", 23:06:35 DEBUG "data": { 23:06:35 DEBUG "resultType": "vector", 23:06:35 DEBUG "result": [ 23:06:35 DEBUG { 23:06:35 DEBUG "metric": { 23:06:35 DEBUG "pod": "ds-idrepo-0" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG [...] 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG 23:06:35 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:06:35 DEBUG }, 23:06:35 DEBUG "value": [ 23:06:35 DEBUG 1713650795, 23:06:35 DEBUG "0" 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG ] 23:06:35 DEBUG } 23:06:35 DEBUG } 23:06:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650795" 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650795, 23:06:36 DEBUG "0" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650796" 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650796" 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "pod": "ds-idrepo-0" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "pod": "ds-cts-2" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "0" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650796" 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650796" 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "pod": "ds-idrepo-0" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "0" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "6967873536" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650796" 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650796" 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "pod": "ds-idrepo-0" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "pod": "ds-idrepo-2" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "0" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "pod": "ds-idrepo-0" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "300815363.813181" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650796" 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650796" 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "403425" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650796" 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "instance": "10.142.0.107:9100" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "instance": "10.142.0.43:9100" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "93037482.83333333" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "14.281417850705774" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650796" 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650796" 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650796" 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "pod": "ds-cts-0" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "pod": "ds-cts-2" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "25.5" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "pod": "ds-idrepo-0" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "9.601562825118881" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "2425.8333333333335" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650796" 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "2405.766666666667" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650796" 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "pod": "ds-idrepo-0" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "pod": "ds-idrepo-2" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "0" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650796" 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "instance": "10.142.0.107:9100" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "instance": "10.142.0.43:9100" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "0.42436666666664374" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650796" 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "instance": "10.142.0.107:9100" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "instance": "10.142.0.43:9100" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "93034847.83333333" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650796" 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "137.8995" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650796" 23:06:36 INFO [http_cmd]: http status code OK 23:06:36 DEBUG --- status code --- 23:06:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:06:36 DEBUG --- http response --- 23:06:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:06:36 DEBUG ----- output ----- 23:06:36 DEBUG { 23:06:36 DEBUG "status": "success", 23:06:36 DEBUG "data": { 23:06:36 DEBUG "resultType": "vector", 23:06:36 DEBUG "result": [ 23:06:36 DEBUG { 23:06:36 DEBUG "metric": { 23:06:36 DEBUG "instance": "10.142.0.107:9100" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG [...] 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG 23:06:36 DEBUG "instance": "10.142.0.43:9100" 23:06:36 DEBUG }, 23:06:36 DEBUG "value": [ 23:06:36 DEBUG 1713650796, 23:06:36 DEBUG "4290593661952" 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG ] 23:06:36 DEBUG } 23:06:36 DEBUG } 23:06:37 INFO 23:06:37 INFO [loop_until]: kubectl --namespace=xlou top pods 23:06:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:06:37 INFO [loop_until]: OK (rc = 0) 23:06:37 DEBUG --- stdout --- 23:06:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 17952m 8806Mi am-767f6b67b5-qcqz2 17226m 8953Mi am-767f6b67b5-vbltz 20104m 9000Mi ds-cts-0 20157m 12362Mi ds-cts-1 19340m 12470Mi ds-cts-2 19595m 12582Mi ds-idrepo-0 7509m 3914Mi ds-idrepo-1 7407m 3946Mi ds-idrepo-2 6721m 3888Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 4m 1238Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 14735m 6941Mi 23:06:37 DEBUG --- stderr --- 23:06:37 DEBUG 23:06:37 INFO 23:06:37 INFO [loop_until]: kubectl --namespace=xlou top node 23:06:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:06:38 INFO [loop_until]: OK (rc = 0) 23:06:38 DEBUG --- stdout --- 23:06:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6289Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7082m 23% 7686Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 15937m 53% 11574Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3366Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19849m 66% 15610Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5211Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 17241m 57% 11869Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 15782m 52% 3931Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 18221m 61% 11262Mi 10% gke-xlou-cdm-ds-0f601751-j488 7136m 23% 7311Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 19221m 64% 17503Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 7196m 24% 7924Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19908m 66% 15287Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 15248m 95% 14936Mi 25% 23:06:38 DEBUG --- stderr --- 23:06:38 DEBUG 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650855" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650855" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650855" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650855" 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "data": { 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "result": [ 23:07:35 DEBUG { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "pod": "ds-cts-0" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "pod": "ds-cts-2" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 1713650855, 23:07:35 DEBUG "27.448966666666667" 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "data": { 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "result": [ 23:07:35 DEBUG { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "instance": "10.142.0.107:9100" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "instance": "10.142.0.43:9100" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 1713650855, 23:07:35 DEBUG "0.05399999999999636" 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "data": { 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "result": [ 23:07:35 DEBUG { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 1713650855, 23:07:35 DEBUG "1072.8333333333335" 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "data": { 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "result": [ 23:07:35 DEBUG { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "pod": "ds-idrepo-0" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "pod": "ds-idrepo-2" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 1713650855, 23:07:35 DEBUG "0" 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650855" 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650855" 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "data": { 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "result": [ 23:07:35 DEBUG { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "pod": "ds-idrepo-1" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "pod": "idm-b85f9d978-wnhxb" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 1713650855, 23:07:35 DEBUG "0" 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650855" 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "data": { 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "result": [ 23:07:35 DEBUG { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "pod": "am-767f6b67b5-9zw22" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "pod": "am-767f6b67b5-vbltz" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 1713650855, 23:07:35 DEBUG "2352.1666666666665" 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650855" 23:07:35 INFO [http_cmd]: http status code OK 23:07:35 DEBUG --- status code --- 23:07:35 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:35 DEBUG --- http response --- 23:07:35 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:07:35 DEBUG ----- output ----- 23:07:35 DEBUG { 23:07:35 DEBUG "status": "success", 23:07:35 DEBUG "data": { 23:07:35 DEBUG "resultType": "vector", 23:07:35 DEBUG "result": [ 23:07:35 DEBUG { 23:07:35 DEBUG "metric": { 23:07:35 DEBUG "pod": "ds-cts-0" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG [...] 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG 23:07:35 DEBUG "pod": "ds-cts-2" 23:07:35 DEBUG }, 23:07:35 DEBUG "value": [ 23:07:35 DEBUG 1713650855, 23:07:35 DEBUG "0" 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG ] 23:07:35 DEBUG } 23:07:35 DEBUG } 23:07:35 ERROR [http_cmd]: ERROR 23:07:35 ERROR --- status code --- 23:07:35 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:07:35 ERROR --- http response --- 23:07:35 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:07:35 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650856" 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "0" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650856" 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650856" 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650856" 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "pod": "ds-idrepo-0" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "pod": "ds-cts-2" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "0" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "11066449920" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650856" 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650856" 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "pod": "ds-idrepo-0" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "pod": "overseer-0-967d957f7-zk66v" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "0" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "14.77863325070958" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650856" 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650856" 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "pod": "ds-idrepo-0" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "pod": "ds-idrepo-2" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "0" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "pod": "ds-idrepo-0" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "0" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650856" 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "453108.5" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650856" 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "instance": "10.142.0.107:9100" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "instance": "10.142.0.43:9100" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "56429094.333333336" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650856" 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650856" 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "pod": "ds-idrepo-0" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "0" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "pod": "ds-cts-0" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "pod": "ds-cts-2" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "31.066666666666666" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650856" 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "2352.133333333333" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650856" 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650856" 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "2352.4333333333334" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650856" 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650856" 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "instance": "10.142.0.107:9100" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "instance": "10.142.0.43:9100" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "56429090.13333333" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "pod": "ds-idrepo-0" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "pod": "ds-idrepo-2" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "0" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "instance": "10.142.0.107:9100" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "instance": "10.142.0.43:9100" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "0.37599999999994693" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650856" 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "140.6205" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650856" 23:07:36 INFO [http_cmd]: http status code OK 23:07:36 DEBUG --- status code --- 23:07:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:07:36 DEBUG --- http response --- 23:07:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:07:36 DEBUG ----- output ----- 23:07:36 DEBUG { 23:07:36 DEBUG "status": "success", 23:07:36 DEBUG "data": { 23:07:36 DEBUG "resultType": "vector", 23:07:36 DEBUG "result": [ 23:07:36 DEBUG { 23:07:36 DEBUG "metric": { 23:07:36 DEBUG "instance": "10.142.0.107:9100" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG [...] 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG 23:07:36 DEBUG "instance": "10.142.0.43:9100" 23:07:36 DEBUG }, 23:07:36 DEBUG "value": [ 23:07:36 DEBUG 1713650856, 23:07:36 DEBUG "4294672692224" 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG ] 23:07:36 DEBUG } 23:07:36 DEBUG } 23:07:37 INFO 23:07:37 INFO [loop_until]: kubectl --namespace=xlou top pods 23:07:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:07:37 INFO [loop_until]: OK (rc = 0) 23:07:37 DEBUG --- stdout --- 23:07:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 13506m 8795Mi am-767f6b67b5-qcqz2 12589m 8914Mi am-767f6b67b5-vbltz 14821m 8986Mi ds-cts-0 19217m 12354Mi ds-cts-1 18231m 12480Mi ds-cts-2 18835m 12588Mi ds-idrepo-0 5419m 3914Mi ds-idrepo-1 5932m 3947Mi ds-idrepo-2 5461m 3889Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 14648m 11157Mi 23:07:37 DEBUG --- stderr --- 23:07:37 DEBUG 23:07:38 INFO 23:07:38 INFO [loop_until]: kubectl --namespace=xlou top node 23:07:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:07:38 INFO [loop_until]: OK (rc = 0) 23:07:38 DEBUG --- stdout --- 23:07:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 6293Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5495m 18% 7683Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12281m 41% 11554Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3364Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19026m 63% 15600Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 5217Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 13167m 44% 11850Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 14551m 48% 3937Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 13975m 46% 11248Mi 10% gke-xlou-cdm-ds-0f601751-j488 5762m 19% 7304Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18035m 60% 17487Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 5902m 19% 7937Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19175m 64% 15294Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 15141m 95% 19439Mi 33% 23:07:38 DEBUG --- stderr --- 23:07:38 DEBUG 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650915" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650915" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650915" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650915" 23:08:35 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650915" 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG --- http response --- 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG { 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG { 23:08:36 DEBUG { 23:08:36 DEBUG { 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG "data": { 23:08:36 DEBUG "data": { 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG { 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG "result": [ 23:08:36 DEBUG "result": [ 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG { 23:08:36 DEBUG { 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "data": { 23:08:36 DEBUG "data": { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "pod": "ds-idrepo-0" 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG "pod": "ds-cts-0" 23:08:36 DEBUG "pod": "ds-idrepo-1" 23:08:36 DEBUG }, 23:08:36 DEBUG "result": [ 23:08:36 DEBUG "result": [ 23:08:36 DEBUG }, 23:08:36 DEBUG { 23:08:36 DEBUG }, 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG { 23:08:36 DEBUG { 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG 23:08:36 DEBUG "data": { 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "ds-cts-0" 23:08:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:08:36 DEBUG 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG }, 23:08:36 DEBUG }, 23:08:36 DEBUG 23:08:36 DEBUG "result": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "value": [ 23:08:36 DEBUG [...] 23:08:36 DEBUG { 23:08:36 DEBUG [...] 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "metric": { 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "instance": "10.142.0.107:9100" 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG }, 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "ds-idrepo-2" 23:08:36 DEBUG [...] 23:08:36 DEBUG [...] 23:08:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "pod": "ds-cts-2" 23:08:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:08:36 DEBUG }, 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG }, 23:08:36 DEBUG 23:08:36 DEBUG }, 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 1713650915, 23:08:36 DEBUG 23:08:36 DEBUG 1713650915, 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "0" 23:08:36 DEBUG "pod": "ds-cts-2" 23:08:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:08:36 DEBUG "3760.366666666667" 23:08:36 DEBUG [...] 23:08:36 DEBUG "0.20193333333333308" 23:08:36 DEBUG "0" 23:08:36 DEBUG ] 23:08:36 DEBUG }, 23:08:36 DEBUG }, 23:08:36 DEBUG ] 23:08:36 DEBUG 23:08:36 DEBUG ] 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG "value": [ 23:08:36 DEBUG "value": [ 23:08:36 DEBUG } 23:08:36 DEBUG 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG 1713650915, 23:08:36 DEBUG 1713650915, 23:08:36 DEBUG ] 23:08:36 DEBUG 23:08:36 DEBUG ] 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG "0" 23:08:36 DEBUG "2120.733333333333" 23:08:36 DEBUG } 23:08:36 DEBUG "instance": "10.142.0.43:9100" 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG }, 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG "value": [ 23:08:36 DEBUG ] 23:08:36 DEBUG ] 23:08:36 DEBUG 1713650915, 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG "0.1123333333333335" 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650916" 23:08:36 ERROR [http_cmd]: ERROR 23:08:36 ERROR --- status code --- 23:08:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:08:36 ERROR --- http response --- 23:08:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:08:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "0" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650916" 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "pod": "ds-idrepo-0" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "ds-cts-2" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "0" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "15151538176" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650916" 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "pod": "ds-idrepo-1" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "488.4913536076327" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650916" 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "14.771145376684228" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG { 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "437693" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG { 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "instance": "10.142.0.107:9100" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "instance": "10.142.0.43:9100" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "80154752.13333333" 23:08:36 DEBUG ] 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG } 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG ] 23:08:36 DEBUG { 23:08:36 DEBUG } 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG } 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "pod": "ds-idrepo-0" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "ds-idrepo-2" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "0" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "pod": "ds-idrepo-1" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "218722.00357781752" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "pod": "ds-idrepo-1" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "16.93120900417354" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "pod": "ds-cts-0" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "ds-cts-2" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "27" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650916" 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "instance": "10.142.0.107:9100" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "instance": "10.142.0.43:9100" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "0.41279999999999384" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "2120.9666666666667" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "pod": "ds-idrepo-0" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "ds-idrepo-2" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "0" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "2120.266666666667" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650916" 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "instance": "10.142.0.107:9100" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "instance": "10.142.0.43:9100" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "80154589.66666666" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650916" 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "142.9975" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650916" 23:08:36 INFO [http_cmd]: http status code OK 23:08:36 DEBUG --- status code --- 23:08:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:08:36 DEBUG --- http response --- 23:08:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:08:36 DEBUG ----- output ----- 23:08:36 DEBUG { 23:08:36 DEBUG "status": "success", 23:08:36 DEBUG "data": { 23:08:36 DEBUG "resultType": "vector", 23:08:36 DEBUG "result": [ 23:08:36 DEBUG { 23:08:36 DEBUG "metric": { 23:08:36 DEBUG "instance": "10.142.0.107:9100" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG [...] 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG 23:08:36 DEBUG "instance": "10.142.0.43:9100" 23:08:36 DEBUG }, 23:08:36 DEBUG "value": [ 23:08:36 DEBUG 1713650916, 23:08:36 DEBUG "4299204350976" 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG ] 23:08:36 DEBUG } 23:08:36 DEBUG } 23:08:37 INFO 23:08:37 INFO [loop_until]: kubectl --namespace=xlou top pods 23:08:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:37 INFO [loop_until]: OK (rc = 0) 23:08:37 DEBUG --- stdout --- 23:08:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 14592m 8797Mi am-767f6b67b5-qcqz2 16003m 8921Mi am-767f6b67b5-vbltz 17556m 8986Mi ds-cts-0 19452m 12356Mi ds-cts-1 18583m 12470Mi ds-cts-2 20472m 12582Mi ds-idrepo-0 5896m 3914Mi ds-idrepo-1 6402m 3947Mi ds-idrepo-2 6138m 3905Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 14856m 14980Mi 23:08:37 DEBUG --- stderr --- 23:08:37 DEBUG 23:08:38 INFO 23:08:38 INFO [loop_until]: kubectl --namespace=xlou top node 23:08:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:08:38 INFO [loop_until]: OK (rc = 0) 23:08:38 DEBUG --- stdout --- 23:08:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6288Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 6107m 20% 7675Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13938m 46% 11555Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3370Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 19610m 65% 15578Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5193Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 15588m 52% 11850Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 13330m 44% 3975Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 15457m 51% 11247Mi 10% gke-xlou-cdm-ds-0f601751-j488 6141m 20% 7334Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18765m 62% 17464Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 6191m 20% 7944Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 20386m 68% 15280Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 15152m 95% 22840Mi 38% 23:08:38 DEBUG --- stderr --- 23:08:38 DEBUG 23:09:23 WARNING File /tmp/lodemon_data-20240420_223134/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [20/Apr/2024 23:09:23] "GET /monitoring/average?start_time=20240420_220550&stop_time=20240420_220723 HTTP/1.1" 200 - 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "ds-cts-0" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "ds-cts-2" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "0" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650976" 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "ds-idrepo-0" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "ds-idrepo-2" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "0" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "ds-cts-0" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "ds-cts-2" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "27.54203333333333" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG { 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "data": { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:36 DEBUG "pod": "ds-idrepo-1" 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG }, 23:09:36 DEBUG "result": [ 23:09:36 DEBUG }, 23:09:36 DEBUG { 23:09:36 DEBUG "value": [ 23:09:36 DEBUG { 23:09:36 DEBUG "value": [ 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG 23:09:36 DEBUG "metric": { 23:09:36 DEBUG 23:09:36 DEBUG "data": { 23:09:36 DEBUG 23:09:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:09:36 DEBUG 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG 23:09:36 DEBUG }, 23:09:36 DEBUG 23:09:36 DEBUG "result": [ 23:09:36 DEBUG [...] 23:09:36 DEBUG "value": [ 23:09:36 DEBUG [...] 23:09:36 DEBUG { 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "metric": { 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "instance": "10.142.0.107:9100" 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG }, 23:09:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:09:36 DEBUG [...] 23:09:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:09:36 DEBUG "value": [ 23:09:36 DEBUG }, 23:09:36 DEBUG 23:09:36 DEBUG }, 23:09:36 DEBUG 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG 23:09:36 DEBUG "0" 23:09:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:09:36 DEBUG "282.3333333333333" 23:09:36 DEBUG [...] 23:09:36 DEBUG ] 23:09:36 DEBUG }, 23:09:36 DEBUG ] 23:09:36 DEBUG 23:09:36 DEBUG } 23:09:36 DEBUG "value": [ 23:09:36 DEBUG } 23:09:36 DEBUG 23:09:36 DEBUG ] 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG ] 23:09:36 DEBUG 23:09:36 DEBUG } 23:09:36 DEBUG "2928.733333333333" 23:09:36 DEBUG } 23:09:36 DEBUG "instance": "10.142.0.43:9100" 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG }, 23:09:36 DEBUG } 23:09:36 DEBUG "value": [ 23:09:36 DEBUG ] 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG } 23:09:36 DEBUG "0.07533333333332543" 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713650976" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 ERROR [http_cmd]: ERROR 23:09:36 ERROR --- status code --- 23:09:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:09:36 ERROR --- http response --- 23:09:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:09:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "0" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713650976" 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "17248018432" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "ds-idrepo-1" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "315768537.18645555" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "ds-idrepo-0" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "ds-idrepo-2" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "0" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "ds-idrepo-0" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "ds-cts-2" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "0" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "ds-idrepo-1" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "0" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713650976" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650976" 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "14.680984209144775" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "335654" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "instance": "10.142.0.107:9100" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "instance": "10.142.0.43:9100" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "48891326.06666666" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "ds-idrepo-1" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "10.890559164105996" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "ds-cts-0" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "ds-cts-2" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "32.86666666666667" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650976" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "ds-idrepo-0" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "ds-idrepo-2" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "0" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "instance": "10.142.0.107:9100" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "instance": "10.142.0.43:9100" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "0.4500666666667342" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "283" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "279.6666666666667" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650976" 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "instance": "10.142.0.107:9100" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "instance": "10.142.0.43:9100" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "48892210.1" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713650976" 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "145.3595" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713650976" 23:09:36 INFO [http_cmd]: http status code OK 23:09:36 DEBUG --- status code --- 23:09:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:09:36 DEBUG --- http response --- 23:09:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:09:36 DEBUG ----- output ----- 23:09:36 DEBUG { 23:09:36 DEBUG "status": "success", 23:09:36 DEBUG "data": { 23:09:36 DEBUG "resultType": "vector", 23:09:36 DEBUG "result": [ 23:09:36 DEBUG { 23:09:36 DEBUG "metric": { 23:09:36 DEBUG "instance": "10.142.0.107:9100" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG [...] 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG 23:09:36 DEBUG "instance": "10.142.0.43:9100" 23:09:36 DEBUG }, 23:09:36 DEBUG "value": [ 23:09:36 DEBUG 1713650976, 23:09:36 DEBUG "4303672058880" 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG ] 23:09:36 DEBUG } 23:09:36 DEBUG } 23:09:37 INFO 23:09:37 INFO [loop_until]: kubectl --namespace=xlou top pods 23:09:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:09:37 INFO [loop_until]: OK (rc = 0) 23:09:37 DEBUG --- stdout --- 23:09:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 9763m 8797Mi am-767f6b67b5-qcqz2 6700m 8919Mi am-767f6b67b5-vbltz 9370m 8777Mi ds-cts-0 18546m 12353Mi ds-cts-1 18208m 12471Mi ds-cts-2 19457m 12578Mi ds-idrepo-0 779m 3914Mi ds-idrepo-1 924m 3946Mi ds-idrepo-2 966m 3905Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1178Mi idm-b85f9d978-wnhxb 3m 1238Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-967d957f7-zk66v 5328m 17Mi 23:09:37 DEBUG --- stderr --- 23:09:37 DEBUG 23:09:38 INFO 23:09:38 INFO [loop_until]: kubectl --namespace=xlou top node 23:09:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:09:38 INFO [loop_until]: OK (rc = 0) 23:09:38 DEBUG --- stdout --- 23:09:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 71m 0% 6280Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 744m 2% 7682Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2894m 9% 11580Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3369Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18438m 61% 15596Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 5185Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 3809m 12% 11877Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 1505m 5% 3588Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 3136m 10% 11074Mi 9% gke-xlou-cdm-ds-0f601751-j488 687m 2% 7321Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 18230m 61% 17483Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 791m 2% 7931Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19373m 64% 15280Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 2177m 13% 7960Mi 13% 23:09:38 DEBUG --- stderr --- 23:09:38 DEBUG 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651036" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG { 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG "data": { 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "result": [ 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG { 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "data": { 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "pod": "ds-idrepo-0" 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG "result": [ 23:10:36 DEBUG }, 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG { 23:10:36 DEBUG "value": [ 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "metric": { 23:10:36 DEBUG 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "data": { 23:10:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:10:36 DEBUG 23:10:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG }, 23:10:36 DEBUG 23:10:36 DEBUG }, 23:10:36 DEBUG "result": [ 23:10:36 DEBUG "value": [ 23:10:36 DEBUG [...] 23:10:36 DEBUG "value": [ 23:10:36 DEBUG { 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "metric": { 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "instance": "10.142.0.107:9100" 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG }, 23:10:36 DEBUG [...] 23:10:36 DEBUG "pod": "ds-idrepo-2" 23:10:36 DEBUG [...] 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG }, 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:10:36 DEBUG "0.06666666666666667" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:10:36 DEBUG [...] 23:10:36 DEBUG }, 23:10:36 DEBUG ] 23:10:36 DEBUG }, 23:10:36 DEBUG 23:10:36 DEBUG "value": [ 23:10:36 DEBUG } 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG ] 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG 23:10:36 DEBUG "0.1" 23:10:36 DEBUG } 23:10:36 DEBUG "2683.9333333333334" 23:10:36 DEBUG "instance": "10.142.0.43:9100" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG }, 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG "value": [ 23:10:36 DEBUG ] 23:10:36 DEBUG ] 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG "0.06466666666666658" 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "ds-cts-0" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "ds-cts-2" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "0" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "ds-cts-0" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "ds-cts-2" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "24.52926666666667" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "ds-idrepo-0" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "ds-idrepo-1" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "0" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 ERROR [http_cmd]: ERROR 23:10:36 ERROR --- status code --- 23:10:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:10:36 ERROR --- http response --- 23:10:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:10:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "0" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "116371456" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "217316.5" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "ds-idrepo-0" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "0" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "ds-idrepo-0" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "ds-idrepo-2" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "0.06666666666666667" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "ds-idrepo-0" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "ds-cts-2" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "0" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "ds-idrepo-0" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "ds-idrepo-1" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "11244.723331431831" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713651036" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "0.03426875392962761" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "instance": "10.142.0.107:9100" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "instance": "10.142.0.43:9100" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "18025176.333333332" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG { 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "ds-cts-0" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "ds-cts-2" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "29.5" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "ds-idrepo-0" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "0" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "0.1" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651036" 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "ds-idrepo-0" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "ds-idrepo-2" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "0.06666666666666667" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "0.03333333333333333" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "instance": "10.142.0.107:9100" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "instance": "10.142.0.43:9100" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "0.4360666666666636" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "instance": "10.142.0.107:9100" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "instance": "10.142.0.43:9100" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "18024914.7" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "147.29700000000003" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651036" 23:10:36 INFO [http_cmd]: http status code OK 23:10:36 DEBUG --- status code --- 23:10:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:10:36 DEBUG --- http response --- 23:10:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:10:36 DEBUG ----- output ----- 23:10:36 DEBUG { 23:10:36 DEBUG "status": "success", 23:10:36 DEBUG "data": { 23:10:36 DEBUG "resultType": "vector", 23:10:36 DEBUG "result": [ 23:10:36 DEBUG { 23:10:36 DEBUG "metric": { 23:10:36 DEBUG "instance": "10.142.0.107:9100" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG [...] 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG 23:10:36 DEBUG "instance": "10.142.0.43:9100" 23:10:36 DEBUG }, 23:10:36 DEBUG "value": [ 23:10:36 DEBUG 1713651036, 23:10:36 DEBUG "4307057658880" 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG ] 23:10:36 DEBUG } 23:10:36 DEBUG } 23:10:37 INFO 23:10:37 INFO [loop_until]: kubectl --namespace=xlou top pods 23:10:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:10:37 INFO [loop_until]: OK (rc = 0) 23:10:37 DEBUG --- stdout --- 23:10:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 2690m 8722Mi am-767f6b67b5-qcqz2 2892m 8551Mi am-767f6b67b5-vbltz 2926m 8620Mi ds-cts-0 19462m 12349Mi ds-cts-1 17582m 12473Mi ds-cts-2 19554m 12567Mi ds-idrepo-0 1167m 3916Mi ds-idrepo-1 1054m 4063Mi ds-idrepo-2 1037m 3933Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 1180Mi idm-b85f9d978-wnhxb 5m 1239Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 978m 348Mi 23:10:37 DEBUG --- stderr --- 23:10:37 DEBUG 23:10:38 INFO 23:10:38 INFO [loop_until]: kubectl --namespace=xlou top node 23:10:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:10:38 INFO [loop_until]: OK (rc = 0) 23:10:38 DEBUG --- stdout --- 23:10:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6298Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1386m 4% 7635Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2736m 9% 11493Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 91m 0% 3389Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18670m 62% 15575Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 199m 0% 5210Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2957m 9% 11491Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 112m 0% 3515Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 3063m 10% 10897Mi 9% gke-xlou-cdm-ds-0f601751-j488 1233m 4% 7312Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 17650m 59% 17438Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 1293m 4% 7906Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19332m 64% 15279Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1284m 8% 8326Mi 14% 23:10:38 DEBUG --- stderr --- 23:10:38 DEBUG 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651096" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "0" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "instance": "10.142.0.107:9100" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "instance": "10.142.0.43:9100" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "0.1183333333333337" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG { 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG "data": { 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG "result": [ 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG { 23:11:36 DEBUG { 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "pod": "ds-idrepo-0" 23:11:36 DEBUG "pod": "ds-cts-0" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 DEBUG "data": { 23:11:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:11:36 DEBUG }, 23:11:36 DEBUG }, 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG "value": [ 23:11:36 DEBUG "result": [ 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG { 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "metric": { 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG [...] 23:11:36 DEBUG }, 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "ds-idrepo-2" 23:11:36 DEBUG "pod": "ds-cts-2" 23:11:36 DEBUG 23:11:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:11:36 DEBUG }, 23:11:36 DEBUG }, 23:11:36 DEBUG [...] 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "63.06666666666666" 23:11:36 DEBUG "23.092733333333335" 23:11:36 DEBUG 23:11:36 DEBUG "0" 23:11:36 DEBUG ] 23:11:36 DEBUG ] 23:11:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG }, 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG ] 23:11:36 DEBUG "value": [ 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG "2789.8999999999996" 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713651096" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "pod": "ds-cts-0" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "ds-cts-2" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "0" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 ERROR [http_cmd]: ERROR 23:11:36 ERROR --- status code --- 23:11:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:11:36 ERROR --- http response --- 23:11:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:11:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "pod": "ds-idrepo-0" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "overseer-0-967d957f7-zk66v" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "0" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713651096" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "746545152" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "93708" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651096" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "pod": "ds-idrepo-0" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "overseer-0-967d957f7-zk66v" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "32259343.263677895" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "pod": "ds-idrepo-0" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "ds-idrepo-2" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "6.7" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "pod": "ds-idrepo-0" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "overseer-0-967d957f7-zk66v" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "0" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "instance": "10.142.0.107:9100" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "instance": "10.142.0.43:9100" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "19547862.666666664" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "pod": "ds-idrepo-0" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "ds-cts-2" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "0" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713651096" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "1.1306441757303094" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651096" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (98 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG "result": [ 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG { 23:11:36 DEBUG { 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG { 23:11:36 DEBUG "pod": "ds-cts-0" 23:11:36 DEBUG "pod": "ds-idrepo-0" 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG }, 23:11:36 DEBUG }, 23:11:36 DEBUG "data": { 23:11:36 DEBUG "value": [ 23:11:36 DEBUG "value": [ 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "result": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG { 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "metric": { 23:11:36 DEBUG [...] 23:11:36 DEBUG [...] 23:11:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG }, 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "ds-cts-2" 23:11:36 DEBUG "pod": "overseer-0-967d957f7-zk66v" 23:11:36 DEBUG 23:11:36 DEBUG }, 23:11:36 DEBUG }, 23:11:36 DEBUG 23:11:36 DEBUG "value": [ 23:11:36 DEBUG "value": [ 23:11:36 DEBUG [...] 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG 23:11:36 DEBUG "29.566666666666666" 23:11:36 DEBUG "2.0115071933156603" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG 23:11:36 DEBUG ] 23:11:36 DEBUG ] 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:11:36 DEBUG ] 23:11:36 DEBUG ] 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG }, 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG "value": [ 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG "0" 23:11:36 DEBUG { 23:11:36 DEBUG ] 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG } 23:11:36 DEBUG "data": { 23:11:36 DEBUG ] 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG } 23:11:36 DEBUG "result": [ 23:11:36 DEBUG } 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "instance": "10.142.0.107:9100" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "instance": "10.142.0.43:9100" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "0.42223333333342905" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651096" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "pod": "ds-idrepo-0" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "ds-idrepo-2" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "8.9" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG { 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "0" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "instance": "10.142.0.107:9100" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "instance": "10.142.0.43:9100" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "19548073.166666668" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713651096" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "148.928" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651096" 23:11:36 INFO [http_cmd]: http status code OK 23:11:36 DEBUG --- status code --- 23:11:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:11:36 DEBUG --- http response --- 23:11:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:11:36 DEBUG ----- output ----- 23:11:36 DEBUG { 23:11:36 DEBUG "status": "success", 23:11:36 DEBUG "data": { 23:11:36 DEBUG "resultType": "vector", 23:11:36 DEBUG "result": [ 23:11:36 DEBUG { 23:11:36 DEBUG "metric": { 23:11:36 DEBUG "instance": "10.142.0.107:9100" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG [...] 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG 23:11:36 DEBUG "instance": "10.142.0.43:9100" 23:11:36 DEBUG }, 23:11:36 DEBUG "value": [ 23:11:36 DEBUG 1713651096, 23:11:36 DEBUG "4310358531072" 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG ] 23:11:36 DEBUG } 23:11:36 DEBUG } 23:11:37 INFO 23:11:37 INFO [loop_until]: kubectl --namespace=xlou top pods 23:11:37 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:11:37 INFO [loop_until]: OK (rc = 0) 23:11:37 DEBUG --- stdout --- 23:11:37 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 2628m 8718Mi am-767f6b67b5-qcqz2 2731m 8551Mi am-767f6b67b5-vbltz 2902m 8624Mi ds-cts-0 18509m 12346Mi ds-cts-1 16951m 12341Mi ds-cts-2 19725m 12572Mi ds-idrepo-0 579m 3916Mi ds-idrepo-1 681m 3967Mi ds-idrepo-2 620m 3926Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1180Mi idm-b85f9d978-wnhxb 4m 1239Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 1001m 1123Mi 23:11:37 DEBUG --- stderr --- 23:11:37 DEBUG 23:11:38 INFO 23:11:38 INFO [loop_until]: kubectl --namespace=xlou top node 23:11:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:11:38 INFO [loop_until]: OK (rc = 0) 23:11:38 DEBUG --- stdout --- 23:11:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 6292Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 616m 2% 7683Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2889m 9% 11496Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3384Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 18258m 61% 15558Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 147m 0% 5183Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2795m 9% 11476Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 80m 0% 3446Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 2689m 9% 10898Mi 9% gke-xlou-cdm-ds-0f601751-j488 586m 1% 7342Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 15512m 51% 17374Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 620m 2% 7950Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 19286m 64% 15256Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1076m 6% 9196Mi 15% 23:11:38 DEBUG --- stderr --- 23:11:38 DEBUG 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651156" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713651156" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 ERROR [http_cmd]: ERROR 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 ERROR --- status code --- 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 ERROR --- http response --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG { 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG "data": { 23:12:36 DEBUG "data": { 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG "result": [ 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:12:36 DEBUG { 23:12:36 DEBUG "pod": "ds-cts-0" 23:12:36 DEBUG }, 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "data": { 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "result": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG { 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG "metric": { 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG "instance": "10.142.0.107:9100" 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG }, 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:12:36 DEBUG 23:12:36 DEBUG "pod": "ds-cts-2" 23:12:36 DEBUG }, 23:12:36 DEBUG 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG [...] 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "0" 23:12:36 DEBUG 23:12:36 DEBUG "0" 23:12:36 DEBUG ] 23:12:36 DEBUG 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG "instance": "10.142.0.43:9100" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG }, 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG "value": [ 23:12:36 DEBUG } 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "0.06766666666666386" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "0" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "ds-cts-0" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "ds-cts-2" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "18.0033" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "ds-idrepo-0" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "ds-idrepo-2" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "0" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "466.26666666666665" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (89 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "ds-idrepo-0" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "ds-cts-2" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "85903196.398623" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713651156" 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "4383" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "1830744064" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "ds-idrepo-0" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "ds-idrepo-2" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "0" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651156" 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "ds-idrepo-0" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "ds-idrepo-2" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "0" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (89 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "ds-idrepo-0" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "ds-cts-2" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "70704357.31309031" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "instance": "10.142.0.107:9100" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "instance": "10.142.0.43:9100" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "8598922.266666666" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "ds-idrepo-0" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "ds-cts-2" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "0" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713651156" 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "1.0383969261861286" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "ds-cts-0" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "ds-cts-2" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "31.733333333333334" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651156" 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "instance": "10.142.0.107:9100" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "instance": "10.142.0.43:9100" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "8600617.566666666" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651156" 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG "result": [ 23:12:36 DEBUG "result": [ 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG { 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "instance": "10.142.0.107:9100" 23:12:36 DEBUG "pod": "ds-idrepo-0" 23:12:36 DEBUG "data": { 23:12:36 DEBUG }, 23:12:36 DEBUG }, 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "result": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG { 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "metric": { 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:12:36 DEBUG [...] 23:12:36 DEBUG [...] 23:12:36 DEBUG }, 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "instance": "10.142.0.43:9100" 23:12:36 DEBUG "pod": "ds-idrepo-2" 23:12:36 DEBUG 23:12:36 DEBUG }, 23:12:36 DEBUG }, 23:12:36 DEBUG [...] 23:12:36 DEBUG "value": [ 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG 23:12:36 DEBUG "0.11000000000002974" 23:12:36 DEBUG "0" 23:12:36 DEBUG 23:12:36 DEBUG ] 23:12:36 DEBUG ] 23:12:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG }, 23:12:36 DEBUG ] 23:12:36 DEBUG ] 23:12:36 DEBUG "value": [ 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 DEBUG "0" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "ds-idrepo-0" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "ds-idrepo-2" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "0" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "0" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713651156" 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "149.762" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651156" 23:12:36 INFO [http_cmd]: http status code OK 23:12:36 DEBUG --- status code --- 23:12:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:12:36 DEBUG --- http response --- 23:12:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:12:36 DEBUG ----- output ----- 23:12:36 DEBUG { 23:12:36 DEBUG "status": "success", 23:12:36 DEBUG "data": { 23:12:36 DEBUG "resultType": "vector", 23:12:36 DEBUG "result": [ 23:12:36 DEBUG { 23:12:36 DEBUG "metric": { 23:12:36 DEBUG "instance": "10.142.0.107:9100" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG [...] 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG 23:12:36 DEBUG "instance": "10.142.0.43:9100" 23:12:36 DEBUG }, 23:12:36 DEBUG "value": [ 23:12:36 DEBUG 1713651156, 23:12:36 DEBUG "4312269624320" 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG ] 23:12:36 DEBUG } 23:12:36 DEBUG } 23:12:38 INFO 23:12:38 INFO [loop_until]: kubectl --namespace=xlou top pods 23:12:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:12:38 INFO [loop_until]: OK (rc = 0) 23:12:38 DEBUG --- stdout --- 23:12:38 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 1094m 8717Mi am-767f6b67b5-qcqz2 605m 8544Mi am-767f6b67b5-vbltz 522m 8619Mi ds-cts-0 18054m 12341Mi ds-cts-1 1724m 12332Mi ds-cts-2 17795m 12569Mi ds-idrepo-0 14m 3916Mi ds-idrepo-1 87m 3967Mi ds-idrepo-2 14m 3925Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 995m 2116Mi 23:12:38 DEBUG --- stderr --- 23:12:38 DEBUG 23:12:38 INFO 23:12:38 INFO [loop_until]: kubectl --namespace=xlou top node 23:12:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:12:38 INFO [loop_until]: OK (rc = 0) 23:12:38 DEBUG --- stdout --- 23:12:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6293Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 72m 0% 7681Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1179m 3% 11445Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3376Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 17664m 59% 15538Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5172Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 730m 2% 11456Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 79m 0% 3411Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 520m 1% 10885Mi 9% gke-xlou-cdm-ds-0f601751-j488 65m 0% 7330Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 673m 2% 17370Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 91m 0% 7951Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 17875m 59% 15235Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 1044m 6% 9956Mi 16% 23:12:38 DEBUG --- stderr --- 23:12:38 DEBUG 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "ds-cts-0" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "ds-cts-2" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713651216" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651216" 23:13:36 ERROR [http_cmd]: ERROR 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 ERROR --- status code --- 23:13:36 DEBUG --- status code --- 23:13:36 ERROR http status code is 400 (expected [200, 401, 403, 404]) 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 ERROR --- http response --- 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 23:13:36 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "instance": "10.142.0.107:9100" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "instance": "10.142.0.43:9100" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0.0006666666666660604" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "ds-cts-0" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "ds-cts-2" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "6.7644" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "ds-idrepo-1" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "ds-idrepo-0" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "ds-idrepo-2" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0.03333333333333333" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0.5" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "2409816064" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "ds-idrepo-0" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "ds-idrepo-2" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651216" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "instance": "10.142.0.107:9100" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "instance": "10.142.0.43:9100" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "27808.066666666666" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "ds-idrepo-1" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "286733646.3352127" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "ds-idrepo-1" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "524.8590466427473" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "ds-idrepo-0" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "ds-cts-2" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0.8964791322053148" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "instance": "10.142.0.107:9100" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "instance": "10.142.0.43:9100" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0.0011999999999270964" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "ds-idrepo-1" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-767f6b67b5-qcqz2" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0.5972513774972456" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "ds-cts-0" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "ds-cts-2" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "29.466666666666665" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "ds-idrepo-0" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "ds-idrepo-2" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "instance": "10.142.0.107:9100" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "instance": "10.142.0.43:9100" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "29141.666666666664" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "0" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "pod": "am-767f6b67b5-9zw22" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "pod": "am-767f6b67b5-vbltz" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "149.871" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:36 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713651216" 23:13:36 INFO [http_cmd]: http status code OK 23:13:36 DEBUG --- status code --- 23:13:36 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 23:13:36 DEBUG --- http response --- 23:13:36 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 23:13:36 DEBUG ----- output ----- 23:13:36 DEBUG { 23:13:36 DEBUG "status": "success", 23:13:36 DEBUG "data": { 23:13:36 DEBUG "resultType": "vector", 23:13:36 DEBUG "result": [ 23:13:36 DEBUG { 23:13:36 DEBUG "metric": { 23:13:36 DEBUG "instance": "10.142.0.107:9100" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG [...] 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG 23:13:36 DEBUG "instance": "10.142.0.43:9100" 23:13:36 DEBUG }, 23:13:36 DEBUG "value": [ 23:13:36 DEBUG 1713651216, 23:13:36 DEBUG "4312343102464" 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG ] 23:13:36 DEBUG } 23:13:36 DEBUG } 23:13:38 INFO 23:13:38 INFO [loop_until]: kubectl --namespace=xlou top pods 23:13:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:13:38 INFO [loop_until]: OK (rc = 0) 23:13:38 DEBUG --- stdout --- 23:13:38 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 648m 8709Mi am-767f6b67b5-qcqz2 595m 8534Mi am-767f6b67b5-vbltz 592m 8607Mi ds-cts-0 16952m 8496Mi ds-cts-1 17m 12333Mi ds-cts-2 17191m 11897Mi ds-idrepo-0 14m 3915Mi ds-idrepo-1 13m 3967Mi ds-idrepo-2 13m 3925Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1179Mi idm-b85f9d978-wnhxb 3m 1239Mi lodemon-5497cb87d7-mlld4 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-967d957f7-zk66v 783m 2964Mi 23:13:38 DEBUG --- stderr --- 23:13:38 DEBUG 23:13:38 INFO 23:13:38 INFO [loop_until]: kubectl --namespace=xlou top node 23:13:38 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 23:13:38 INFO [loop_until]: OK (rc = 0) 23:13:38 DEBUG --- stdout --- 23:13:38 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6297Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 74m 0% 7685Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 745m 2% 11444Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3363Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 17232m 57% 13338Mi 11% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5173Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 726m 2% 11444Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 81m 0% 3373Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 710m 2% 10875Mi 9% gke-xlou-cdm-ds-0f601751-j488 68m 0% 7335Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 72m 0% 17359Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 97m 0% 7944Mi 7% gke-xlou-cdm-ds-0f601751-nmxm 17660m 59% 14566Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 668m 4% 10575Mi 18% 23:13:38 DEBUG --- stderr --- 23:13:38 DEBUG 23:13:49 INFO Finished: True 23:13:49 INFO Waiting for threads to register finish flag 23:14:38 INFO Done. Have a nice day! :) 127.0.0.1 - - [20/Apr/2024 23:14:38] "GET /monitoring/stop HTTP/1.1" 200 - 23:14:41 WARNING File /tmp/lodemon_data-20240420_223134/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6521', ...) 127.0.0.1 - - [20/Apr/2024 23:14:45] "GET /monitoring/process HTTP/1.1" 200 -