==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-68f967fd98-vkx4v Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Fri, 19 Apr 2024 19:29:17 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=68f967fd98 skaffold.dev/run-id=69cbb141-6663-480a-ad57-a12e9334593c Annotations: Status: Running IP: 10.106.46.168 IPs: IP: 10.106.46.168 Controlled By: ReplicaSet/lodemon-68f967fd98 Containers: lodemon: Container ID: containerd://bbb8c14869c0286505255c59cbb352a765c6c3940834edc9468cb3b06f97dd84 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:03f76e6326d713f4d73dc474e0e25fbbbaecc7c1d8c0590bfd94d68ac0d98c97 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Fri, 19 Apr 2024 19:29:18 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ftzxt (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-ftzxt: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 12m default-scheduler Successfully assigned xlou/lodemon-68f967fd98-vkx4v to gke-xlou-cdm-ds-0f601751-mc5d Normal Pulling 12m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 12m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 344.00173ms (344.00711ms including waiting) Normal Created 12m kubelet Created container lodemon Normal Started 12m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.46.168:8080 Press CTRL+C to quit 20:30:24 INFO 20:30:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:24 INFO [loop_until]: OK (rc = 0) 20:30:24 DEBUG --- stdout --- 20:30:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:24 DEBUG --- stderr --- 20:30:24 DEBUG 20:30:24 INFO 20:30:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:24 INFO [loop_until]: OK (rc = 0) 20:30:24 DEBUG --- stdout --- 20:30:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:24 DEBUG --- stderr --- 20:30:24 DEBUG 20:30:24 INFO 20:30:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:24 INFO [loop_until]: OK (rc = 0) 20:30:24 DEBUG --- stdout --- 20:30:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:24 DEBUG --- stderr --- 20:30:24 DEBUG 20:30:24 INFO 20:30:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:25 INFO [loop_until]: OK (rc = 0) 20:30:25 DEBUG --- stdout --- 20:30:25 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:25 DEBUG --- stderr --- 20:30:25 DEBUG 20:30:25 INFO 20:30:25 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:25 INFO [loop_until]: OK (rc = 0) 20:30:25 DEBUG --- stdout --- 20:30:25 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:25 DEBUG --- stderr --- 20:30:25 DEBUG 20:30:25 INFO 20:30:25 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:25 INFO [loop_until]: OK (rc = 0) 20:30:25 DEBUG --- stdout --- 20:30:25 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:25 DEBUG --- stderr --- 20:30:25 DEBUG 20:30:25 INFO 20:30:25 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:25 INFO [loop_until]: OK (rc = 0) 20:30:25 DEBUG --- stdout --- 20:30:25 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:25 DEBUG --- stderr --- 20:30:25 DEBUG 20:30:25 INFO 20:30:25 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:25 INFO [loop_until]: OK (rc = 0) 20:30:25 DEBUG --- stdout --- 20:30:25 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:25 DEBUG --- stderr --- 20:30:25 DEBUG 20:30:25 INFO 20:30:25 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:25 INFO [loop_until]: OK (rc = 0) 20:30:25 DEBUG --- stdout --- 20:30:25 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:25 DEBUG --- stderr --- 20:30:25 DEBUG 20:30:25 INFO 20:30:25 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:25 INFO [loop_until]: OK (rc = 0) 20:30:25 DEBUG --- stdout --- 20:30:25 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:25 DEBUG --- stderr --- 20:30:25 DEBUG 20:30:25 INFO 20:30:25 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:26 INFO [loop_until]: OK (rc = 0) 20:30:26 DEBUG --- stdout --- 20:30:26 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:26 DEBUG --- stderr --- 20:30:26 DEBUG 20:30:26 INFO 20:30:26 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:26 INFO [loop_until]: OK (rc = 0) 20:30:26 DEBUG --- stdout --- 20:30:26 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:26 DEBUG --- stderr --- 20:30:26 DEBUG 20:30:26 INFO 20:30:26 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:26 INFO [loop_until]: OK (rc = 0) 20:30:26 DEBUG --- stdout --- 20:30:26 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:26 DEBUG --- stderr --- 20:30:26 DEBUG 20:30:26 INFO 20:30:26 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:26 INFO [loop_until]: OK (rc = 0) 20:30:26 DEBUG --- stdout --- 20:30:26 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:26 DEBUG --- stderr --- 20:30:26 DEBUG 20:30:26 INFO 20:30:26 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:26 INFO [loop_until]: OK (rc = 0) 20:30:26 DEBUG --- stdout --- 20:30:26 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:26 DEBUG --- stderr --- 20:30:26 DEBUG 20:30:26 INFO 20:30:26 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:26 INFO [loop_until]: OK (rc = 0) 20:30:26 DEBUG --- stdout --- 20:30:26 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:26 DEBUG --- stderr --- 20:30:26 DEBUG 20:30:26 INFO 20:30:26 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:26 INFO [loop_until]: OK (rc = 0) 20:30:26 DEBUG --- stdout --- 20:30:26 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:26 DEBUG --- stderr --- 20:30:26 DEBUG 20:30:26 INFO 20:30:26 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:26 INFO [loop_until]: OK (rc = 0) 20:30:26 DEBUG --- stdout --- 20:30:26 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:26 DEBUG --- stderr --- 20:30:26 DEBUG 20:30:27 INFO 20:30:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:27 INFO [loop_until]: OK (rc = 0) 20:30:27 DEBUG --- stdout --- 20:30:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:27 DEBUG --- stderr --- 20:30:27 DEBUG 20:30:27 INFO 20:30:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:27 INFO [loop_until]: OK (rc = 0) 20:30:27 DEBUG --- stdout --- 20:30:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:27 DEBUG --- stderr --- 20:30:27 DEBUG 20:30:27 INFO 20:30:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:27 INFO [loop_until]: OK (rc = 0) 20:30:27 DEBUG --- stdout --- 20:30:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:27 DEBUG --- stderr --- 20:30:27 DEBUG 20:30:27 INFO 20:30:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:27 INFO [loop_until]: OK (rc = 0) 20:30:27 DEBUG --- stdout --- 20:30:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:27 DEBUG --- stderr --- 20:30:27 DEBUG 20:30:27 INFO 20:30:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:27 INFO [loop_until]: OK (rc = 0) 20:30:27 DEBUG --- stdout --- 20:30:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:27 DEBUG --- stderr --- 20:30:27 DEBUG 20:30:27 INFO 20:30:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:27 INFO [loop_until]: OK (rc = 0) 20:30:27 DEBUG --- stdout --- 20:30:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:27 DEBUG --- stderr --- 20:30:27 DEBUG 20:30:27 INFO 20:30:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:27 INFO [loop_until]: OK (rc = 0) 20:30:27 DEBUG --- stdout --- 20:30:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:27 DEBUG --- stderr --- 20:30:27 DEBUG 20:30:28 INFO 20:30:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:28 INFO [loop_until]: OK (rc = 0) 20:30:28 DEBUG --- stdout --- 20:30:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:28 DEBUG --- stderr --- 20:30:28 DEBUG 20:30:28 INFO 20:30:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 20:30:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:28 INFO [loop_until]: OK (rc = 0) 20:30:28 DEBUG --- stdout --- 20:30:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 20:30:28 DEBUG --- stderr --- 20:30:28 DEBUG 20:30:28 INFO Initializing monitoring instance threads 20:30:28 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 20:30:28 INFO Starting instance threads 20:30:28 INFO 20:30:28 INFO Thread started 20:30:28 INFO [loop_until]: kubectl --namespace=xlou top node 20:30:28 INFO 20:30:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:28 INFO Thread started 20:30:28 INFO [loop_until]: kubectl --namespace=xlou top pods 20:30:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG --- status code --- 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 DEBUG --- status code --- 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 DEBUG --- status code --- 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 DEBUG --- status code --- 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 INFO Thread started 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG --- http response --- 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 DEBUG --- http response --- 20:30:28 INFO Thread started 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (161 lines): 20:30:28 DEBUG ----- output ----- 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 DEBUG --- http response --- 20:30:28 INFO Thread started 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (161 lines): 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (161 lines): 20:30:28 DEBUG { 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (161 lines): 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG ----- output ----- 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 DEBUG ----- output ----- 20:30:28 INFO Thread started 20:30:28 DEBUG { 20:30:28 INFO [loop_until]: OK (rc = 0) 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG { 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 DEBUG ----- output ----- 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG { 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG { 20:30:28 DEBUG "data": { 20:30:28 DEBUG { 20:30:28 DEBUG { 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 DEBUG "status": "success", 20:30:28 INFO Thread started 20:30:28 DEBUG --- stdout --- 20:30:28 DEBUG { 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG { 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG { 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG { 20:30:28 DEBUG { 20:30:28 DEBUG { 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG "data": { 20:30:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7m 8956Mi am-767f6b67b5-qcqz2 7m 8755Mi am-767f6b67b5-vbltz 8m 8939Mi ds-cts-0 14m 10932Mi ds-cts-1 11m 11190Mi ds-cts-2 13m 10957Mi ds-idrepo-0 14m 3715Mi ds-idrepo-1 31m 3560Mi ds-idrepo-2 13m 3494Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1174Mi idm-b85f9d978-wnhxb 3m 1218Mi lodemon-68f967fd98-vkx4v 87m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1m 5895Mi 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 DEBUG "status": "success", 20:30:28 INFO Thread started 20:30:28 DEBUG "data": { 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG "data": { 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG "data": { 20:30:28 DEBUG "result": [ 20:30:28 DEBUG "data": { 20:30:28 DEBUG "data": { 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG --- stderr --- 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG "data": { 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 DEBUG "resultType": "vector", 20:30:28 INFO Thread started 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG "data": { 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG "data": { 20:30:28 DEBUG "resultType": "vector", 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 DEBUG "data": { 20:30:28 DEBUG "data": { 20:30:28 DEBUG "data": { 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG { 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:28 DEBUG 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG "result": [ 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 INFO [loop_until]: OK (rc = 0) 20:30:28 DEBUG { 20:30:28 INFO Thread started 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:28 DEBUG "result": [ 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG "result": [ 20:30:28 DEBUG "metric": { 20:30:28 DEBUG "result": [ 20:30:28 DEBUG "result": [ 20:30:28 DEBUG "result": [ 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG "result": [ 20:30:28 DEBUG { 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG --- stdout --- 20:30:28 DEBUG "status": "success", 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713555028" 20:30:28 DEBUG "result": [ 20:30:28 INFO Thread started 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG "result": [ 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG { 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG "result": [ 20:30:28 DEBUG "result": [ 20:30:28 DEBUG "result": [ 20:30:28 DEBUG { 20:30:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:30:28 DEBUG { 20:30:28 DEBUG { 20:30:28 DEBUG { 20:30:28 DEBUG { 20:30:28 DEBUG { 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:28 DEBUG "metric": { 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 6183Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 73m 0% 7302Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 56m 0% 11657Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3384Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 71m 0% 13927Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5192Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 69m 0% 11628Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 77m 0% 2942Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 11131Mi 9% gke-xlou-cdm-ds-0f601751-j488 67m 0% 6706Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 112m 0% 15965Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 122m 0% 7420Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 71m 0% 13366Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 68m 0% 9620Mi 16% 20:30:28 DEBUG "data": { 20:30:28 DEBUG { 20:30:28 DEBUG "metric": { 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 DEBUG { 20:30:28 DEBUG { 20:30:28 DEBUG { 20:30:28 DEBUG "metric": { 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG { 20:30:28 DEBUG { 20:30:28 DEBUG "metric": { 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 DEBUG { 20:30:28 DEBUG "metric": { 20:30:28 DEBUG }, 20:30:28 DEBUG "metric": { 20:30:28 DEBUG "metric": { 20:30:28 DEBUG "metric": { 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG "metric": { 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG { 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG --- stderr --- 20:30:28 DEBUG "resultType": "vector", 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 DEBUG "instance": "10.142.0.107:9100" 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG "metric": { 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG "metric": { 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:30:28 ERROR [http_cmd]: ERROR 20:30:28 DEBUG "pod": "ds-idrepo-0" 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG "metric": { 20:30:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "instance": "10.142.0.107:9100" 20:30:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:30:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:30:28 DEBUG "data": { 20:30:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG "instance": "10.142.0.107:9100" 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:30:28 DEBUG "result": [ 20:30:28 DEBUG 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG }, 20:30:28 DEBUG "data": { 20:30:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555028" 20:30:28 INFO Thread started 20:30:28 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:30:28 DEBUG }, 20:30:28 DEBUG }, 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:30:28 DEBUG ----- output ----- 20:30:28 ERROR --- status code --- 20:30:28 DEBUG }, 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG "pod": "ds-cts-0" 20:30:28 DEBUG }, 20:30:28 DEBUG 20:30:28 DEBUG }, 20:30:28 DEBUG }, 20:30:28 DEBUG }, 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG "result": [ 20:30:28 DEBUG }, 20:30:28 DEBUG }, 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG { 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "resultType": "vector", 20:30:28 INFO All threads has been started 127.0.0.1 - - [19/Apr/2024 20:30:28] "GET /monitoring/start HTTP/1.1" 200 - 20:30:28 DEBUG "data": { 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG "value": [ 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:28 DEBUG }, 20:30:28 DEBUG { 20:30:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:30:28 DEBUG "value": [ 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG }, 20:30:28 DEBUG "value": [ 20:30:28 DEBUG 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "data": { 20:30:28 DEBUG { 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "value": [ 20:30:28 DEBUG { 20:30:28 DEBUG "metric": { 20:30:28 DEBUG --- http response --- 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 DEBUG 20:30:28 DEBUG "result": [ 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "result": [ 20:30:28 DEBUG 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "status": "success", 20:30:28 ERROR --- http response --- 20:30:28 DEBUG 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:30:28 DEBUG "value": [ 20:30:28 DEBUG 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG "metric": { 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG 20:30:28 DEBUG { 20:30:28 DEBUG 20:30:28 DEBUG { 20:30:28 DEBUG 20:30:28 DEBUG { 20:30:28 DEBUG 20:30:28 DEBUG "data": { 20:30:28 DEBUG 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG [...] 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG "result": [ 20:30:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:30:28 DEBUG 20:30:28 DEBUG "pod": "ds-idrepo-0" 20:30:28 DEBUG 20:30:28 DEBUG "data": { 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG 20:30:28 DEBUG "metric": { 20:30:28 DEBUG 20:30:28 DEBUG "metric": { 20:30:28 DEBUG 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG 20:30:28 DEBUG { 20:30:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 INFO [http_cmd]: http status code OK 20:30:28 DEBUG { 20:30:28 DEBUG }, 20:30:28 DEBUG 20:30:28 DEBUG }, 20:30:28 DEBUG [...] 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG { 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG [...] 20:30:28 DEBUG "pod": "ds-idrepo-0" 20:30:28 DEBUG 20:30:28 DEBUG [...] 20:30:28 DEBUG [...] 20:30:28 DEBUG "data": { 20:30:28 DEBUG 20:30:28 DEBUG [...] 20:30:28 DEBUG [...] 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG 20:30:28 DEBUG [...] 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG 20:30:28 DEBUG [...] 20:30:28 DEBUG [...] 20:30:28 DEBUG [...] 20:30:28 DEBUG --- status code --- 20:30:28 DEBUG "metric": { 20:30:28 DEBUG "value": [ 20:30:28 DEBUG [...] 20:30:28 DEBUG "value": [ 20:30:28 DEBUG 20:30:28 DEBUG "result": [ 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:28 DEBUG }, 20:30:28 DEBUG 20:30:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:30:28 DEBUG 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG "result": [ 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG "data": { 20:30:28 DEBUG [...] 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:30:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG { 20:30:28 DEBUG "data": { 20:30:28 DEBUG 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG "value": [ 20:30:28 DEBUG 20:30:28 DEBUG }, 20:30:28 DEBUG 20:30:28 DEBUG "result": [ 20:30:28 DEBUG { 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG "pod": "lodemon-f8459c597-6tj6f" 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG --- http response --- 20:30:28 DEBUG }, 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG "metric": { 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG 20:30:28 DEBUG { 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG "value": [ 20:30:28 DEBUG 20:30:28 DEBUG { 20:30:28 DEBUG "metric": { 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG "result": [ 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG }, 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG "value": [ 20:30:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:30:28 DEBUG 20:30:28 DEBUG [...] 20:30:28 DEBUG 20:30:28 DEBUG "instance": "10.142.0.43:9100" 20:30:28 DEBUG "pod": "ds-cts-0" 20:30:28 DEBUG "result": [ 20:30:28 DEBUG "instance": "10.142.0.43:9100" 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG "data": { 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:30:28 DEBUG "metric": { 20:30:28 DEBUG "pod": "ds-cts-0" 20:30:28 DEBUG }, 20:30:28 DEBUG "pod": "ds-idrepo-2" 20:30:28 DEBUG { 20:30:28 DEBUG 20:30:28 DEBUG "pod": "lodemon-f8459c597-6tj6f" 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "instance": "10.142.0.43:9100" 20:30:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:30:28 DEBUG "pod": "lodemon-f8459c597-6tj6f" 20:30:28 DEBUG 20:30:28 DEBUG ----- output ----- 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG "pod": "lodemon-f8459c597-6tj6f" 20:30:28 DEBUG }, 20:30:28 DEBUG }, 20:30:28 DEBUG { 20:30:28 DEBUG }, 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:30:28 DEBUG }, 20:30:28 DEBUG 20:30:28 DEBUG }, 20:30:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:30:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:30:28 DEBUG "value": [ 20:30:28 DEBUG }, 20:30:28 DEBUG "metric": { 20:30:28 DEBUG "pod": "ds-idrepo-0" 20:30:28 DEBUG "pod": "ds-cts-2" 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG }, 20:30:28 DEBUG }, 20:30:28 DEBUG }, 20:30:28 DEBUG 20:30:28 DEBUG { 20:30:28 DEBUG [...] 20:30:28 DEBUG 20:30:28 DEBUG }, 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "metric": { 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "result": [ 20:30:28 DEBUG 20:30:28 DEBUG "value": [ 20:30:28 DEBUG 20:30:28 DEBUG "value": [ 20:30:28 DEBUG }, 20:30:28 DEBUG }, 20:30:28 DEBUG 20:30:28 DEBUG "value": [ 20:30:28 DEBUG }, 20:30:28 DEBUG }, 20:30:28 DEBUG }, 20:30:28 DEBUG "0" 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "value": [ 20:30:28 DEBUG 20:30:28 DEBUG "status": "success", 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG "value": [ 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG 20:30:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG { 20:30:28 DEBUG [...] 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG [...] 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "value": [ 20:30:28 DEBUG 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "value": [ 20:30:28 DEBUG ] 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG [...] 20:30:28 DEBUG "data": { 20:30:28 DEBUG 20:30:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG "0.002000000000000076" 20:30:28 DEBUG 20:30:28 DEBUG }, 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "metric": { 20:30:28 DEBUG 20:30:28 DEBUG "0.031233551473741482" 20:30:28 DEBUG 20:30:28 DEBUG "6278918144" 20:30:28 DEBUG 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG 20:30:28 DEBUG "0" 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG } 20:30:28 DEBUG "28173.333333333332" 20:30:28 DEBUG "0" 20:30:28 DEBUG "0" 20:30:28 DEBUG 20:30:28 DEBUG "resultType": "vector", 20:30:28 DEBUG 20:30:28 DEBUG }, 20:30:28 DEBUG "0.000825639833832077" 20:30:28 DEBUG ] 20:30:28 DEBUG 20:30:28 DEBUG "27109.733333333334" 20:30:28 DEBUG 20:30:28 DEBUG "instance": "10.142.0.107:9100" 20:30:28 DEBUG 20:30:28 DEBUG ] 20:30:28 DEBUG 20:30:28 DEBUG ] 20:30:28 DEBUG 20:30:28 DEBUG "0" 20:30:28 DEBUG [...] 20:30:28 DEBUG ] 20:30:28 DEBUG "7020.422392005346" 20:30:28 DEBUG 20:30:28 DEBUG "0" 20:30:28 DEBUG ] 20:30:28 DEBUG ] 20:30:28 DEBUG ] 20:30:28 DEBUG ] 20:30:28 DEBUG 20:30:28 DEBUG "result": [ 20:30:28 DEBUG "pod": "ds-idrepo-2" 20:30:28 DEBUG "value": [ 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG [...] 20:30:28 DEBUG ] 20:30:28 DEBUG 20:30:28 DEBUG }, 20:30:28 DEBUG 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG ] 20:30:28 DEBUG 20:30:28 DEBUG } 20:30:28 DEBUG ] 20:30:28 DEBUG 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG { 20:30:28 DEBUG }, 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG } 20:30:28 DEBUG ] 20:30:28 DEBUG 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "pod": "ds-idrepo-2" 20:30:28 DEBUG ] 20:30:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:30:28 DEBUG ] 20:30:28 DEBUG [...] 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG [...] 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG ] 20:30:28 DEBUG ] 20:30:28 DEBUG ] 20:30:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:30:28 DEBUG "metric": { 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "0" 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG ] 20:30:28 DEBUG [...] 20:30:28 DEBUG 20:30:28 DEBUG }, 20:30:28 DEBUG } 20:30:28 DEBUG }, 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG ] 20:30:28 DEBUG 20:30:28 DEBUG } 20:30:28 DEBUG ] 20:30:28 DEBUG 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG "instance": "10.142.0.107:9100" 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG "value": [ 20:30:28 DEBUG } 20:30:28 DEBUG "value": [ 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG "pod": "ds-cts-2" 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG }, 20:30:28 DEBUG }, 20:30:28 DEBUG "value": [ 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG "pod": "ds-cts-2" 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG 20:30:28 DEBUG "0" 20:30:28 DEBUG } 20:30:28 DEBUG }, 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG } 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "0" 20:30:28 DEBUG 20:30:28 DEBUG ] 20:30:28 DEBUG }, 20:30:28 DEBUG 20:30:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:30:28 DEBUG "0" 20:30:28 DEBUG 20:30:28 DEBUG ] 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "pod": "ds-cts-2" 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG ] 20:30:28 DEBUG 20:30:28 DEBUG } 20:30:28 DEBUG "value": [ 20:30:28 DEBUG [...] 20:30:28 DEBUG }, 20:30:28 DEBUG ] 20:30:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:30:28 DEBUG } 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG }, 20:30:28 DEBUG "0" 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG } 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG 20:30:28 DEBUG "value": [ 20:30:28 DEBUG } 20:30:28 DEBUG }, 20:30:28 DEBUG ] 20:30:28 DEBUG "0" 20:30:28 DEBUG "value": [ 20:30:28 DEBUG ] 20:30:28 DEBUG ] 20:30:28 DEBUG [...] 20:30:28 DEBUG "0" 20:30:28 DEBUG 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG ] 20:30:28 DEBUG "value": [ 20:30:28 DEBUG } 20:30:28 DEBUG ] 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG ] 20:30:28 DEBUG 20:30:28 DEBUG "0.02" 20:30:28 DEBUG } 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG "0" 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG } 20:30:28 DEBUG "instance": "10.142.0.43:9100" 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG "1" 20:30:28 DEBUG ] 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG 20:30:28 DEBUG ] 20:30:28 DEBUG }, 20:30:28 DEBUG } 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG "instance": "10.142.0.43:9100" 20:30:28 DEBUG } 20:30:28 DEBUG "value": [ 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG ] 20:30:28 DEBUG }, 20:30:28 DEBUG } 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG } 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG "value": [ 20:30:28 DEBUG "2063585307648" 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG 1713555028, 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG "0.0036333333333449977" 20:30:28 DEBUG } 20:30:28 DEBUG ] 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG ] 20:30:28 DEBUG } 20:30:28 DEBUG } 20:30:28 DEBUG } 20:31:28 INFO 20:31:28 INFO [loop_until]: kubectl --namespace=xlou top pods 20:31:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO 20:31:28 INFO [loop_until]: kubectl --namespace=xlou top node 20:31:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG { 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "data": { 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "result": [ 20:31:28 DEBUG { 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "instance": "10.142.0.107:9100" 20:31:28 DEBUG }, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG "instance": "10.142.0.43:9100" 20:31:28 DEBUG }, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG "0.01563333333339756" 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG { 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "data": { 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "result": [ 20:31:28 DEBUG { 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:31:28 DEBUG }, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:31:28 DEBUG }, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG "0" 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 INFO [loop_until]: OK (rc = 0) 20:31:28 DEBUG --- stdout --- 20:31:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 3383m 8965Mi am-767f6b67b5-qcqz2 4484m 8766Mi am-767f6b67b5-vbltz 4698m 8956Mi ds-cts-0 4211m 10955Mi ds-cts-1 3025m 11216Mi ds-cts-2 3208m 10978Mi ds-idrepo-0 3942m 3716Mi ds-idrepo-1 3253m 3561Mi ds-idrepo-2 3472m 3495Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1174Mi idm-b85f9d978-wnhxb 3m 1218Mi lodemon-68f967fd98-vkx4v 2m 61Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 5164m 6664Mi 20:31:28 DEBUG --- stderr --- 20:31:28 DEBUG 20:31:28 ERROR [http_cmd]: ERROR 20:31:28 ERROR --- status code --- 20:31:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:31:28 ERROR --- http response --- 20:31:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:31:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713555088" 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 DEBUG --- status code --- 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555088" 20:31:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555088" 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 DEBUG --- status code --- 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 DEBUG --- status code --- 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 INFO [loop_until]: OK (rc = 0) 20:31:28 DEBUG --- status code --- 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG --- status code --- 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 DEBUG --- status code --- 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG --- status code --- 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG --- stdout --- 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 6187Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3765m 12% 7323Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6580m 22% 10660Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3391Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 4093m 13% 13996Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5174Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 5739m 19% 11644Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 6059m 20% 2972Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 6569m 22% 11177Mi 10% gke-xlou-cdm-ds-0f601751-j488 3146m 10% 6746Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 4204m 14% 16071Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3429m 11% 7447Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 3919m 13% 13438Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 4903m 30% 10050Mi 17% 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 INFO [http_cmd]: http status code OK 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG --- stderr --- 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG --- status code --- 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:28 DEBUG 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:31:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:31:28 DEBUG { 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG { 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG --- http response --- 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG { 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG { 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG { 20:31:28 DEBUG "data": { 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "data": { 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "data": { 20:31:28 DEBUG "data": { 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG { 20:31:28 DEBUG "data": { 20:31:28 DEBUG "data": { 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG ----- output ----- 20:31:28 DEBUG "data": { 20:31:28 DEBUG "data": { 20:31:28 DEBUG "data": { 20:31:28 DEBUG "data": { 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "data": { 20:31:28 DEBUG "data": { 20:31:28 DEBUG "data": { 20:31:28 DEBUG "data": { 20:31:28 DEBUG "data": { 20:31:28 DEBUG "data": { 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "data": { 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "data": { 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "data": { 20:31:28 DEBUG "data": { 20:31:28 DEBUG { 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "data": { 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "data": { 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "status": "success", 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG { 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "result": [ 20:31:28 DEBUG { 20:31:28 DEBUG "result": [ 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "data": { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "metric": { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG "metric": { 20:31:28 DEBUG { 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "metric": { 20:31:28 DEBUG { 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "metric": { 20:31:28 DEBUG { 20:31:28 DEBUG { 20:31:28 DEBUG "resultType": "vector", 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "metric": { 20:31:28 DEBUG { 20:31:28 DEBUG "pod": "ds-cts-0" 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "pod": "ds-cts-0" 20:31:28 DEBUG "pod": "ds-idrepo-0" 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:31:28 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:31:28 DEBUG "metric": { 20:31:28 DEBUG { 20:31:28 DEBUG "pod": "ds-idrepo-0" 20:31:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "result": [ 20:31:28 DEBUG "instance": "10.142.0.107:9100" 20:31:28 DEBUG "instance": "10.142.0.107:9100" 20:31:28 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:31:28 DEBUG "instance": "10.142.0.107:9100" 20:31:28 DEBUG "metric": { 20:31:28 DEBUG }, 20:31:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:31:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:31:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:31:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:31:28 DEBUG "metric": { 20:31:28 DEBUG }, 20:31:28 DEBUG }, 20:31:28 DEBUG "pod": "ds-idrepo-0" 20:31:28 DEBUG }, 20:31:28 DEBUG }, 20:31:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:31:28 DEBUG "metric": { 20:31:28 DEBUG }, 20:31:28 DEBUG }, 20:31:28 DEBUG "pod": "ds-idrepo-0" 20:31:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:31:28 DEBUG { 20:31:28 DEBUG }, 20:31:28 DEBUG }, 20:31:28 DEBUG }, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:31:28 DEBUG "value": [ 20:31:28 DEBUG }, 20:31:28 DEBUG }, 20:31:28 DEBUG }, 20:31:28 DEBUG }, 20:31:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "value": [ 20:31:28 DEBUG }, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "value": [ 20:31:28 DEBUG }, 20:31:28 DEBUG "pod": "ds-cts-0" 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "value": [ 20:31:28 DEBUG }, 20:31:28 DEBUG }, 20:31:28 DEBUG "metric": { 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "value": [ 20:31:28 DEBUG }, 20:31:28 DEBUG 20:31:28 DEBUG }, 20:31:28 DEBUG 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "value": [ 20:31:28 DEBUG }, 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG "value": [ 20:31:28 DEBUG }, 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "instance": "10.142.0.107:9100" 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG }, 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG [...] 20:31:28 DEBUG [...] 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG [...] 20:31:28 DEBUG [...] 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG "pod": "ds-idrepo-2" 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG [...] 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG "pod": "ds-cts-2" 20:31:28 DEBUG 20:31:28 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG "pod": "ds-cts-2" 20:31:28 DEBUG 20:31:28 DEBUG }, 20:31:28 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:31:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:31:28 DEBUG 20:31:28 DEBUG 20:31:28 DEBUG "pod": "ds-idrepo-2" 20:31:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:31:28 DEBUG 20:31:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:31:28 DEBUG 20:31:28 DEBUG "instance": "10.142.0.43:9100" 20:31:28 DEBUG 20:31:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:31:28 DEBUG "instance": "10.142.0.43:9100" 20:31:28 DEBUG 20:31:28 DEBUG }, 20:31:28 DEBUG 20:31:28 DEBUG }, 20:31:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:31:28 DEBUG }, 20:31:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:31:28 DEBUG }, 20:31:28 DEBUG 20:31:28 DEBUG "value": [ 20:31:28 DEBUG }, 20:31:28 DEBUG }, 20:31:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:31:28 DEBUG 20:31:28 DEBUG }, 20:31:28 DEBUG }, 20:31:28 DEBUG "pod": "ds-cts-2" 20:31:28 DEBUG }, 20:31:28 DEBUG "instance": "10.142.0.43:9100" 20:31:28 DEBUG }, 20:31:28 DEBUG 20:31:28 DEBUG }, 20:31:28 DEBUG }, 20:31:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 20:31:28 DEBUG "value": [ 20:31:28 DEBUG }, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "pod": "ds-idrepo-2" 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "value": [ 20:31:28 DEBUG }, 20:31:28 DEBUG "pod": "ds-cts-2" 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "value": [ 20:31:28 DEBUG }, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG }, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "value": [ 20:31:28 DEBUG }, 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG }, 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG }, 20:31:28 DEBUG "0" 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG }, 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG "instance": "10.142.0.43:9100" 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "0" 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "0" 20:31:28 DEBUG }, 20:31:28 DEBUG "0" 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG "0" 20:31:28 DEBUG "value": [ 20:31:28 DEBUG ] 20:31:28 DEBUG "0" 20:31:28 DEBUG "6700834816" 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG "value": [ 20:31:28 DEBUG "0" 20:31:28 DEBUG "1371.9" 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG "20593.5" 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG "11625384.833333332" 20:31:28 DEBUG }, 20:31:28 DEBUG "2.7272382259618437" 20:31:28 DEBUG "0.012000000000002353" 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG ] 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG ] 20:31:28 DEBUG "value": [ 20:31:28 DEBUG ] 20:31:28 DEBUG "0" 20:31:28 DEBUG ] 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG ] 20:31:28 DEBUG "0.02" 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG ] 20:31:28 DEBUG ] 20:31:28 DEBUG "0" 20:31:28 DEBUG ] 20:31:28 DEBUG "11625777.8" 20:31:28 DEBUG ] 20:31:28 DEBUG "value": [ 20:31:28 DEBUG ] 20:31:28 DEBUG ] 20:31:28 DEBUG "1374.3666666666666" 20:31:28 DEBUG } 20:31:28 DEBUG "1373.2333333333333" 20:31:28 DEBUG } 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG "0" 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG "0" 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG 1713555088, 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG ] 20:31:28 DEBUG ] 20:31:28 DEBUG ] 20:31:28 DEBUG "0" 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG ] 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG "2063699059712" 20:31:28 DEBUG ] 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG ] 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:31:28 DEBUG } 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO 20:32:28 INFO [loop_until]: kubectl --namespace=xlou top pods 20:32:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713555148" 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG { 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "data": { 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "result": [ 20:32:28 DEBUG { 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "instance": "10.142.0.107:9100" 20:32:28 DEBUG }, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "instance": "10.142.0.43:9100" 20:32:28 DEBUG }, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "0.033499999999952484" 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG { 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "data": { 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "result": [ 20:32:28 DEBUG { 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:32:28 DEBUG }, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:32:28 DEBUG }, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "0" 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 ERROR [http_cmd]: ERROR 20:32:28 ERROR --- status code --- 20:32:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:32:28 ERROR --- http response --- 20:32:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:32:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:32:28 INFO [loop_until]: OK (rc = 0) 20:32:28 DEBUG --- stdout --- 20:32:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6706m 7959Mi am-767f6b67b5-qcqz2 6913m 8581Mi am-767f6b67b5-vbltz 7573m 7953Mi ds-cts-0 3765m 10972Mi ds-cts-1 3697m 11237Mi ds-cts-2 3980m 11003Mi ds-idrepo-0 3466m 3716Mi ds-idrepo-1 3316m 3560Mi ds-idrepo-2 3362m 3495Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1174Mi idm-b85f9d978-wnhxb 3m 1218Mi lodemon-68f967fd98-vkx4v 2m 61Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7916m 7608Mi 20:32:28 DEBUG --- stderr --- 20:32:28 DEBUG 20:32:28 INFO 20:32:28 INFO [loop_until]: kubectl --namespace=xlou top node 20:32:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- http response --- 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG { 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "data": { 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "result": [ 20:32:28 DEBUG { 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:32:28 DEBUG }, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "pod": "ds-cts-0" 20:32:28 DEBUG }, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "0" 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- http response --- 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG { 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "data": { 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "result": [ 20:32:28 DEBUG { 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "pod": "ds-idrepo-0" 20:32:28 DEBUG }, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "pod": "ds-idrepo-2" 20:32:28 DEBUG }, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "0" 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 DEBUG --- status code --- 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG --- status code --- 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555148" 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG --- status code --- 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG --- status code --- 20:32:28 INFO [loop_until]: OK (rc = 0) 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG --- status code --- 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG --- status code --- 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG --- status code --- 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG --- status code --- 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- stdout --- 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG --- status code --- 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6181Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3492m 11% 7328Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6926m 23% 10656Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3380Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 3827m 12% 14000Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 167m 0% 5186Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 7027m 23% 11459Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 8810m 29% 3714Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 7714m 25% 10180Mi 9% gke-xlou-cdm-ds-0f601751-j488 3654m 12% 6754Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 3712m 12% 16096Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3628m 12% 7454Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 3901m 13% 13473Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 8182m 51% 10976Mi 18% 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:28 INFO [http_cmd]: http status code OK 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- status code --- 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:28 DEBUG --- stderr --- 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:32:28 DEBUG { 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:32:28 DEBUG { 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:28 DEBUG --- http response --- 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:32:28 DEBUG { 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:28 DEBUG { 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:32:28 DEBUG { 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG "data": { 20:32:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG { 20:32:28 DEBUG { 20:32:28 DEBUG { 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG { 20:32:28 DEBUG { 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG { 20:32:28 DEBUG "data": { 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG { 20:32:28 DEBUG { 20:32:28 DEBUG { 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG ----- output ----- 20:32:28 DEBUG { 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG { 20:32:28 DEBUG { 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG { 20:32:28 DEBUG "data": { 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG { 20:32:28 DEBUG "data": { 20:32:28 DEBUG { 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "data": { 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "result": [ 20:32:28 DEBUG { 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "data": { 20:32:28 DEBUG "data": { 20:32:28 DEBUG "data": { 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "data": { 20:32:28 DEBUG "data": { 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "data": { 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "data": { 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "data": { 20:32:28 DEBUG "data": { 20:32:28 DEBUG { 20:32:28 DEBUG "status": "success", 20:32:28 DEBUG "data": { 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "data": { 20:32:28 DEBUG "data": { 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "data": { 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG { 20:32:28 DEBUG "data": { 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "data": { 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "data": { 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "result": [ 20:32:28 DEBUG { 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG { 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG { 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "result": [ 20:32:28 DEBUG { 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "result": [ 20:32:28 DEBUG { 20:32:28 DEBUG "instance": "10.142.0.107:9100" 20:32:28 DEBUG "result": [ 20:32:28 DEBUG { 20:32:28 DEBUG { 20:32:28 DEBUG { 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "metric": { 20:32:28 DEBUG { 20:32:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "result": [ 20:32:28 DEBUG { 20:32:28 DEBUG "metric": { 20:32:28 DEBUG { 20:32:28 DEBUG "resultType": "vector", 20:32:28 DEBUG "metric": { 20:32:28 DEBUG }, 20:32:28 DEBUG { 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "metric": { 20:32:28 DEBUG { 20:32:28 DEBUG { 20:32:28 DEBUG { 20:32:28 DEBUG "pod": "ds-cts-0" 20:32:28 DEBUG { 20:32:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:32:28 DEBUG "metric": { 20:32:28 DEBUG }, 20:32:28 DEBUG { 20:32:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:32:28 DEBUG { 20:32:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "result": [ 20:32:28 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:32:28 DEBUG "value": [ 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "instance": "10.142.0.107:9100" 20:32:28 DEBUG "pod": "ds-cts-0" 20:32:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "pod": "ds-idrepo-0" 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "metric": { 20:32:28 DEBUG }, 20:32:28 DEBUG "pod": "ds-cts-0" 20:32:28 DEBUG "value": [ 20:32:28 DEBUG "metric": { 20:32:28 DEBUG }, 20:32:28 DEBUG "metric": { 20:32:28 DEBUG }, 20:32:28 DEBUG "pod": "ds-idrepo-0" 20:32:28 DEBUG "instance": "10.142.0.107:9100" 20:32:28 DEBUG { 20:32:28 DEBUG }, 20:32:28 DEBUG 20:32:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:32:28 DEBUG }, 20:32:28 DEBUG }, 20:32:28 DEBUG }, 20:32:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:32:28 DEBUG }, 20:32:28 DEBUG }, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG "instance": "10.142.0.107:9100" 20:32:28 DEBUG "value": [ 20:32:28 DEBUG }, 20:32:28 DEBUG 20:32:28 DEBUG "pod": "ds-idrepo-0" 20:32:28 DEBUG "value": [ 20:32:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:32:28 DEBUG "value": [ 20:32:28 DEBUG }, 20:32:28 DEBUG }, 20:32:28 DEBUG "metric": { 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG }, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG "value": [ 20:32:28 DEBUG }, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:32:28 DEBUG 20:32:28 DEBUG }, 20:32:28 DEBUG 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG }, 20:32:28 DEBUG 20:32:28 DEBUG }, 20:32:28 DEBUG 20:32:28 DEBUG "value": [ 20:32:28 DEBUG "value": [ 20:32:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "value": [ 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG }, 20:32:28 DEBUG 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG }, 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "pod": "ds-cts-2" 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:32:28 DEBUG [...] 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "pod": "ds-idrepo-2" 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG }, 20:32:28 DEBUG "instance": "10.142.0.43:9100" 20:32:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:32:28 DEBUG }, 20:32:28 DEBUG "instance": "10.142.0.43:9100" 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "pod": "ds-cts-2" 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:32:28 DEBUG 20:32:28 DEBUG }, 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG }, 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG [...] 20:32:28 DEBUG "value": [ 20:32:28 DEBUG }, 20:32:28 DEBUG 20:32:28 DEBUG "value": [ 20:32:28 DEBUG }, 20:32:28 DEBUG "pod": "ds-cts-2" 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "pod": "ds-idrepo-2" 20:32:28 DEBUG 20:32:28 DEBUG }, 20:32:28 DEBUG "pod": "ds-cts-2" 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG "pod": "ds-cts-0" 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG }, 20:32:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:32:28 DEBUG }, 20:32:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:32:28 DEBUG }, 20:32:28 DEBUG "instance": "10.142.0.43:9100" 20:32:28 DEBUG "value": [ 20:32:28 DEBUG }, 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:32:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:32:28 DEBUG }, 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "instance": "10.142.0.43:9100" 20:32:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:32:28 DEBUG 20:32:28 DEBUG "0" 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:32:28 DEBUG "8.043668418503612" 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG }, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG }, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG "124787" 20:32:28 DEBUG }, 20:32:28 DEBUG }, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG "0" 20:32:28 DEBUG }, 20:32:28 DEBUG }, 20:32:28 DEBUG 20:32:28 DEBUG ] 20:32:28 DEBUG "34975042.46666667" 20:32:28 DEBUG }, 20:32:28 DEBUG ] 20:32:28 DEBUG "34974938" 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "0" 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "8112689152" 20:32:28 DEBUG "1378.2333333333333" 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG ] 20:32:28 DEBUG "value": [ 20:32:28 DEBUG "value": [ 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG ] 20:32:28 DEBUG "value": [ 20:32:28 DEBUG "value": [ 20:32:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG "value": [ 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG "value": [ 20:32:28 DEBUG ] 20:32:28 DEBUG "0" 20:32:28 DEBUG "0" 20:32:28 DEBUG ] 20:32:28 DEBUG ] 20:32:28 DEBUG ] 20:32:28 DEBUG "0" 20:32:28 DEBUG } 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "0" 20:32:28 DEBUG } 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG }, 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG ] 20:32:28 DEBUG }, 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG ] 20:32:28 DEBUG ] 20:32:28 DEBUG "0" 20:32:28 DEBUG "0" 20:32:28 DEBUG ] 20:32:28 DEBUG ] 20:32:28 DEBUG "0.025666666666664163" 20:32:28 DEBUG "0" 20:32:28 DEBUG "value": [ 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG "0.02" 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG "1378.8333333333333" 20:32:28 DEBUG } 20:32:28 DEBUG "value": [ 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG ] 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG "1379.3333333333333" 20:32:28 DEBUG ] 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG ] 20:32:28 DEBUG ] 20:32:28 DEBUG 1713555148, 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG "2065409680384" 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG ] 20:32:28 DEBUG } 20:32:28 DEBUG } 20:32:28 DEBUG } 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555208" 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG { 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "data": { 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "result": [ 20:33:28 DEBUG { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "instance": "10.142.0.107:9100" 20:33:28 DEBUG }, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG "instance": "10.142.0.43:9100" 20:33:28 DEBUG }, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "0.03506666666671663" 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO 20:33:28 INFO [loop_until]: kubectl --namespace=xlou top pods 20:33:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG { 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "data": { 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "result": [ 20:33:28 DEBUG { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "pod": "ds-idrepo-0" 20:33:28 DEBUG }, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:33:28 DEBUG }, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "0" 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 ERROR [http_cmd]: ERROR 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 ERROR --- status code --- 20:33:28 DEBUG --- status code --- 20:33:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 ERROR --- http response --- 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:33:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:33:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:33:28 INFO [loop_until]: OK (rc = 0) 20:33:28 DEBUG --- stdout --- 20:33:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 6554m 7959Mi am-767f6b67b5-qcqz2 6750m 8582Mi am-767f6b67b5-vbltz 6995m 7951Mi ds-cts-0 3581m 10996Mi ds-cts-1 3309m 11244Mi ds-cts-2 3650m 11002Mi ds-idrepo-0 3053m 3715Mi ds-idrepo-1 3180m 3560Mi ds-idrepo-2 3287m 3494Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1174Mi idm-b85f9d978-wnhxb 4m 1217Mi lodemon-68f967fd98-vkx4v 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8078m 7749Mi 20:33:28 DEBUG --- stderr --- 20:33:28 DEBUG 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG { 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "data": { 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "result": [ 20:33:28 DEBUG { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "pod": "ds-idrepo-0" 20:33:28 DEBUG }, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG "pod": "ds-idrepo-2" 20:33:28 DEBUG }, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "0" 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG { 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "data": { 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "result": [ 20:33:28 DEBUG { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "pod": "ds-idrepo-0" 20:33:28 DEBUG }, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:33:28 DEBUG }, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "1454265.7986090684" 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 INFO 20:33:28 INFO [loop_until]: kubectl --namespace=xlou top node 20:33:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555208" 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG { 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "data": { 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "result": [ 20:33:28 DEBUG { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "instance": "10.142.0.107:9100" 20:33:28 DEBUG }, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG "instance": "10.142.0.43:9100" 20:33:28 DEBUG }, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "0.02533333333333871" 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713555208" 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 DEBUG --- status code --- 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG --- status code --- 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 DEBUG --- http response --- 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 INFO [http_cmd]: http status code OK 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG --- status code --- 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG { 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG { 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG --- http response --- 20:33:28 DEBUG { 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG { 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "data": { 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG "data": { 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG "data": { 20:33:28 DEBUG ----- output ----- 20:33:28 DEBUG { 20:33:28 DEBUG "data": { 20:33:28 DEBUG "data": { 20:33:28 DEBUG "data": { 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "data": { 20:33:28 DEBUG "data": { 20:33:28 DEBUG "data": { 20:33:28 DEBUG "data": { 20:33:28 DEBUG "data": { 20:33:28 DEBUG "data": { 20:33:28 DEBUG "data": { 20:33:28 DEBUG "data": { 20:33:28 DEBUG "data": { 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG { 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG { 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG { 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "data": { 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "status": "success", 20:33:28 DEBUG "data": { 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG { 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "result": [ 20:33:28 DEBUG { 20:33:28 DEBUG "data": { 20:33:28 DEBUG { 20:33:28 DEBUG "data": { 20:33:28 DEBUG { 20:33:28 DEBUG "data": { 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "metric": { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "resultType": "vector", 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG "pod": "ds-idrepo-0" 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "instance": "10.142.0.107:9100" 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:33:28 DEBUG "result": [ 20:33:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:33:28 DEBUG "metric": { 20:33:28 DEBUG { 20:33:28 DEBUG { 20:33:28 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:33:28 DEBUG "instance": "10.142.0.107:9100" 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG }, 20:33:28 DEBUG "pod": "ds-cts-0" 20:33:28 DEBUG "pod": "ds-cts-0" 20:33:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:33:28 DEBUG "pod": "ds-idrepo-0" 20:33:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:33:28 DEBUG "pod": "ds-idrepo-0" 20:33:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:33:28 DEBUG }, 20:33:28 DEBUG { 20:33:28 DEBUG }, 20:33:28 DEBUG { 20:33:28 DEBUG }, 20:33:28 DEBUG "pod": "ds-idrepo-0" 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "metric": { 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:33:28 DEBUG "instance": "10.142.0.107:9100" 20:33:28 DEBUG "value": [ 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "metric": { 20:33:28 DEBUG "value": [ 20:33:28 DEBUG }, 20:33:28 DEBUG "pod": "ds-idrepo-0" 20:33:28 DEBUG "pod": "ds-cts-0" 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "value": [ 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 20:33:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:33:28 DEBUG 20:33:28 DEBUG "pod": "am-767f6b67b5-9zw22" 20:33:28 DEBUG 20:33:28 DEBUG "value": [ 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG }, 20:33:28 DEBUG 20:33:28 DEBUG }, 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG [...] 20:33:28 DEBUG [...] 20:33:28 DEBUG [...] 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG [...] 20:33:28 DEBUG 20:33:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG "pod": "ds-cts-2" 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:33:28 DEBUG "instance": "10.142.0.43:9100" 20:33:28 DEBUG 20:33:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG }, 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:33:28 DEBUG "instance": "10.142.0.43:9100" 20:33:28 DEBUG }, 20:33:28 DEBUG 20:33:28 DEBUG }, 20:33:28 DEBUG "pod": "ds-cts-2" 20:33:28 DEBUG "pod": "ds-cts-2" 20:33:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:33:28 DEBUG "pod": "ds-idrepo-2" 20:33:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:33:28 DEBUG 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG 20:33:28 DEBUG }, 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 20:33:28 DEBUG 20:33:28 DEBUG }, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 20:33:28 DEBUG "instance": "10.142.0.43:9100" 20:33:28 DEBUG "value": [ 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 20:33:28 DEBUG "pod": "ds-idrepo-2" 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:33:28 DEBUG "pod": "ds-cts-2" 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:33:28 DEBUG }, 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG }, 20:33:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "pod": "am-767f6b67b5-vbltz" 20:33:28 DEBUG }, 20:33:28 DEBUG "204313" 20:33:28 DEBUG }, 20:33:28 DEBUG }, 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "2066888870912" 20:33:28 DEBUG }, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "0" 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "0" 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "0" 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "4.13849952638407" 20:33:28 DEBUG "value": [ 20:33:28 DEBUG }, 20:33:28 DEBUG "8.033588253545437" 20:33:28 DEBUG }, 20:33:28 DEBUG "value": [ 20:33:28 DEBUG ] 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "value": [ 20:33:28 DEBUG "8140521472" 20:33:28 DEBUG ] 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG ] 20:33:28 DEBUG "0.000033333333333333335" 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG ] 20:33:28 DEBUG "0" 20:33:28 DEBUG ] 20:33:28 DEBUG "32612760.766666666" 20:33:28 DEBUG ] 20:33:28 DEBUG ] 20:33:28 DEBUG "value": [ 20:33:28 DEBUG ] 20:33:28 DEBUG "value": [ 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG } 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "32614744.633333333" 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG "0" 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG ] 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG } 20:33:28 DEBUG 1713555208, 20:33:28 DEBUG "0" 20:33:28 DEBUG ] 20:33:28 DEBUG "923.416651232046" 20:33:28 DEBUG "0" 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG "1291.6666666666667" 20:33:28 DEBUG ] 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG "1291.2333333333333" 20:33:28 DEBUG ] 20:33:28 DEBUG ] 20:33:28 DEBUG "0.021" 20:33:28 DEBUG ] 20:33:28 DEBUG "1290" 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG ] 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 INFO [loop_until]: OK (rc = 0) 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG ] 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG ] 20:33:28 DEBUG ] 20:33:28 DEBUG --- stdout --- 20:33:28 DEBUG ] 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6196Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3261m 10% 7328Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6525m 21% 10669Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3368Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 3671m 12% 14006Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5178Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 6652m 22% 11447Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 8456m 28% 3741Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 7169m 24% 10182Mi 9% gke-xlou-cdm-ds-0f601751-j488 3473m 11% 6762Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 3490m 11% 16118Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3485m 11% 7455Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 3715m 12% 13476Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 8264m 52% 11101Mi 18% 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG } 20:33:28 DEBUG --- stderr --- 20:33:28 DEBUG } 20:33:28 DEBUG 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "result": [ 20:34:28 DEBUG { 20:34:28 DEBUG "metric": { 20:34:28 DEBUG "instance": "10.142.0.107:9100" 20:34:28 DEBUG }, 20:34:28 DEBUG "value": [ 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG [...] 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG "instance": "10.142.0.43:9100" 20:34:28 DEBUG }, 20:34:28 DEBUG "value": [ 20:34:28 DEBUG 1713555268, 20:34:28 DEBUG "0.03476666666660814" 20:34:28 DEBUG ] 20:34:28 DEBUG } 20:34:28 DEBUG ] 20:34:28 DEBUG } 20:34:28 DEBUG } 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "result": [ 20:34:28 DEBUG { 20:34:28 DEBUG "metric": { 20:34:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:34:28 DEBUG }, 20:34:28 DEBUG "value": [ 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG [...] 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:34:28 DEBUG }, 20:34:28 DEBUG "value": [ 20:34:28 DEBUG 1713555268, 20:34:28 DEBUG "0" 20:34:28 DEBUG ] 20:34:28 DEBUG } 20:34:28 DEBUG ] 20:34:28 DEBUG } 20:34:28 DEBUG } 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713555268" 20:34:28 ERROR [http_cmd]: ERROR 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 DEBUG --- status code --- 20:34:28 ERROR --- status code --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- http response --- 20:34:28 ERROR --- http response --- 20:34:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:34:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:34:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:34:28 INFO 20:34:28 INFO [loop_until]: kubectl --namespace=xlou top pods 20:34:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "result": [ 20:34:28 DEBUG { 20:34:28 DEBUG "metric": { 20:34:28 DEBUG "pod": "ds-idrepo-0" 20:34:28 DEBUG }, 20:34:28 DEBUG "value": [ 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG [...] 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG "pod": "ds-idrepo-2" 20:34:28 DEBUG }, 20:34:28 DEBUG "value": [ 20:34:28 DEBUG 1713555268, 20:34:28 DEBUG "0" 20:34:28 DEBUG ] 20:34:28 DEBUG } 20:34:28 DEBUG ] 20:34:28 DEBUG } 20:34:28 DEBUG } 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "result": [ 20:34:28 DEBUG { 20:34:28 DEBUG "metric": { 20:34:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:34:28 DEBUG }, 20:34:28 DEBUG "value": [ 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG [...] 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:34:28 DEBUG }, 20:34:28 DEBUG "value": [ 20:34:28 DEBUG 1713555268, 20:34:28 DEBUG "0" 20:34:28 DEBUG ] 20:34:28 DEBUG } 20:34:28 DEBUG ] 20:34:28 DEBUG } 20:34:28 DEBUG } 20:34:28 INFO [loop_until]: OK (rc = 0) 20:34:28 DEBUG --- stdout --- 20:34:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 8174m 7973Mi am-767f6b67b5-qcqz2 6861m 8585Mi am-767f6b67b5-vbltz 8346m 7962Mi ds-cts-0 8934m 10999Mi ds-cts-1 6860m 11248Mi ds-cts-2 7493m 11005Mi ds-idrepo-0 3634m 3715Mi ds-idrepo-1 3379m 3561Mi ds-idrepo-2 3615m 3495Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1174Mi idm-b85f9d978-wnhxb 4m 1217Mi lodemon-68f967fd98-vkx4v 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7830m 8066Mi 20:34:28 DEBUG --- stderr --- 20:34:28 DEBUG 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555268" 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "result": [ 20:34:28 DEBUG { 20:34:28 DEBUG "metric": { 20:34:28 DEBUG "instance": "10.142.0.107:9100" 20:34:28 DEBUG }, 20:34:28 DEBUG "value": [ 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG [...] 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG 20:34:28 DEBUG "instance": "10.142.0.43:9100" 20:34:28 DEBUG }, 20:34:28 DEBUG "value": [ 20:34:28 DEBUG 1713555268, 20:34:28 DEBUG "0.023333333333344315" 20:34:28 DEBUG ] 20:34:28 DEBUG } 20:34:28 DEBUG ] 20:34:28 DEBUG } 20:34:28 DEBUG } 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO 20:34:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555268" 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 DEBUG --- status code --- 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 INFO [loop_until]: kubectl --namespace=xlou top node 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 DEBUG --- status code --- 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG --- status code --- 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 DEBUG --- status code --- 20:34:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG --- http response --- 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- http response --- 20:34:28 INFO [http_cmd]: http status code OK 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:34:28 DEBUG --- status code --- 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG { 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG { 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG --- http response --- 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG { 20:34:28 DEBUG { 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:34:28 DEBUG { 20:34:28 DEBUG { 20:34:28 DEBUG { 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG { 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG { 20:34:28 DEBUG "data": { 20:34:28 DEBUG { 20:34:28 DEBUG { 20:34:28 DEBUG "data": { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:34:28 DEBUG { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG { 20:34:28 DEBUG { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "data": { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "data": { 20:34:28 DEBUG { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "data": { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "result": [ 20:34:28 DEBUG "data": { 20:34:28 DEBUG "data": { 20:34:28 DEBUG "result": [ 20:34:28 DEBUG "result": [ 20:34:28 DEBUG ----- output ----- 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "data": { 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "data": { 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "data": { 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG { 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "result": [ 20:34:28 DEBUG { 20:34:28 DEBUG { 20:34:28 DEBUG "result": [ 20:34:28 DEBUG "result": [ 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "result": [ 20:34:28 DEBUG "result": [ 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "result": [ 20:34:28 DEBUG "result": [ 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "result": [ 20:34:28 DEBUG "resultType": "vector", 20:34:28 DEBUG "result": [ 20:34:28 DEBUG "metric": { 20:34:28 DEBUG "result": [ 20:34:28 DEBUG { 20:34:28 DEBUG { 20:34:28 DEBUG "metric": { 20:34:28 DEBUG "status": "success", 20:34:28 DEBUG { 20:34:28 DEBUG { 20:34:28 DEBUG "result": [ 20:34:29 DEBUG { 20:34:28 DEBUG { 20:34:28 DEBUG { 20:34:28 DEBUG "result": [ 20:34:28 DEBUG "result": [ 20:34:28 DEBUG { 20:34:28 DEBUG { 20:34:28 DEBUG "result": [ 20:34:28 DEBUG { 20:34:29 DEBUG "result": [ 20:34:29 DEBUG { 20:34:29 DEBUG "instance": "10.142.0.107:9100" 20:34:29 DEBUG { 20:34:29 DEBUG "metric": { 20:34:29 DEBUG "metric": { 20:34:29 DEBUG "pod": "ds-cts-0" 20:34:29 DEBUG "data": { 20:34:29 DEBUG "metric": { 20:34:29 DEBUG "metric": { 20:34:28 DEBUG "result": [ 20:34:29 DEBUG "metric": { 20:34:29 DEBUG "metric": { 20:34:29 DEBUG "metric": { 20:34:29 DEBUG { 20:34:29 DEBUG { 20:34:29 DEBUG "metric": { 20:34:29 DEBUG "metric": { 20:34:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:34:29 DEBUG "metric": { 20:34:29 DEBUG { 20:34:29 DEBUG "metric": { 20:34:29 DEBUG }, 20:34:29 DEBUG "metric": { 20:34:29 DEBUG "pod": "ds-idrepo-0" 20:34:29 DEBUG "instance": "10.142.0.107:9100" 20:34:29 DEBUG }, 20:34:29 DEBUG "resultType": "vector", 20:34:29 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:34:29 DEBUG "instance": "10.142.0.107:9100" 20:34:29 DEBUG { 20:34:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:34:29 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:34:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:34:29 DEBUG "metric": { 20:34:29 DEBUG "metric": { 20:34:29 DEBUG "pod": "ds-cts-0" 20:34:29 DEBUG { 20:34:29 DEBUG }, 20:34:29 DEBUG "pod": "ds-idrepo-0" 20:34:29 DEBUG "metric": { 20:34:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:34:29 DEBUG }, 20:34:29 DEBUG }, 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "result": [ 20:34:29 DEBUG }, 20:34:29 DEBUG }, 20:34:29 DEBUG "metric": { 20:34:29 DEBUG }, 20:34:29 DEBUG }, 20:34:29 DEBUG }, 20:34:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:34:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:34:29 DEBUG }, 20:34:29 DEBUG "metric": { 20:34:29 DEBUG "value": [ 20:34:29 DEBUG }, 20:34:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:34:29 DEBUG }, 20:34:29 DEBUG 20:34:29 DEBUG }, 20:34:29 DEBUG "value": [ 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG { 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "value": [ 20:34:29 DEBUG }, 20:34:29 DEBUG }, 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "pod": "ds-idrepo-0" 20:34:29 DEBUG 20:34:29 DEBUG "value": [ 20:34:29 DEBUG }, 20:34:29 DEBUG "value": [ 20:34:29 DEBUG 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "value": [ 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG "metric": { 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG }, 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "value": [ 20:34:29 DEBUG 20:34:29 DEBUG }, 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG "value": [ 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG "pod": "ds-cts-0" 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG "value": [ 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG "value": [ 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG [...] 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG [...] 20:34:29 DEBUG [...] 20:34:29 DEBUG }, 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG [...] 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG "value": [ 20:34:29 DEBUG [...] 20:34:29 DEBUG [...] 20:34:29 DEBUG 20:34:29 DEBUG [...] 20:34:29 DEBUG [...] 20:34:29 DEBUG [...] 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG [...] 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG [...] 20:34:29 DEBUG 20:34:29 DEBUG [...] 20:34:29 DEBUG 20:34:29 DEBUG [...] 20:34:29 DEBUG [...] 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG [...] 20:34:29 DEBUG [...] 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG [...] 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG "pod": "ds-cts-2" 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG [...] 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG [...] 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG "instance": "10.142.0.43:9100" 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG "pod": "ds-cts-2" 20:34:29 DEBUG 20:34:29 DEBUG }, 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG }, 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG }, 20:34:29 DEBUG 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:34:29 DEBUG "instance": "10.142.0.43:9100" 20:34:29 DEBUG 20:34:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:34:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:34:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:34:29 DEBUG 20:34:29 DEBUG 20:34:29 DEBUG "pod": "ds-cts-2" 20:34:29 DEBUG 20:34:29 DEBUG }, 20:34:29 DEBUG "pod": "ds-idrepo-2" 20:34:29 DEBUG 20:34:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 20:34:29 DEBUG "instance": "10.142.0.43:9100" 20:34:29 DEBUG "value": [ 20:34:29 DEBUG [...] 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG }, 20:34:29 DEBUG }, 20:34:29 DEBUG 20:34:29 DEBUG }, 20:34:29 DEBUG }, 20:34:29 DEBUG }, 20:34:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:34:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:34:29 DEBUG }, 20:34:29 DEBUG 20:34:29 DEBUG "value": [ 20:34:29 DEBUG }, 20:34:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:34:29 DEBUG }, 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG }, 20:34:29 DEBUG }, 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG 20:34:29 DEBUG "0" 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "value": [ 20:34:29 DEBUG }, 20:34:29 DEBUG }, 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "pod": "ds-idrepo-2" 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG "value": [ 20:34:29 DEBUG }, 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "33585083.63333333" 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "0" 20:34:29 DEBUG ] 20:34:29 DEBUG ] 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG }, 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "value": [ 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG }, 20:34:29 DEBUG "0.021" 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG "value": [ 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG ] 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG "0.00001794693302266695" 20:34:29 DEBUG "33586137.766666666" 20:34:29 DEBUG "value": [ 20:34:29 DEBUG "1246.9333333333334" 20:34:29 DEBUG ] 20:34:29 DEBUG "0" 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG "0.00032855333333333336" 20:34:29 DEBUG "value": [ 20:34:29 DEBUG ] 20:34:29 DEBUG "0" 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG "261383" 20:34:29 DEBUG } 20:34:29 DEBUG "0" 20:34:29 DEBUG "2068269288448" 20:34:29 DEBUG 20:34:29 DEBUG ] 20:34:29 DEBUG ] 20:34:29 DEBUG ] 20:34:29 DEBUG ] 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG "8.049814857821346" 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG "1248.1" 20:34:29 DEBUG ] 20:34:29 DEBUG "1371.4" 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG ] 20:34:29 DEBUG ] 20:34:29 DEBUG ] 20:34:29 DEBUG "pod": "ds-cts-2" 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG "8353505280" 20:34:29 DEBUG ] 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG "0" 20:34:29 DEBUG ] 20:34:29 DEBUG "1246.0333333333333" 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG }, 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG ] 20:34:29 DEBUG "value": [ 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG 1713555268, 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG ] 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG "0" 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 DEBUG ] 20:34:29 DEBUG } 20:34:29 DEBUG } 20:34:29 INFO [loop_until]: OK (rc = 0) 20:34:29 DEBUG --- stdout --- 20:34:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6188Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3700m 12% 7327Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7853m 26% 10672Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3371Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 8531m 28% 13975Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5194Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 7834m 26% 11483Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 7871m 26% 3702Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 8735m 29% 10190Mi 9% gke-xlou-cdm-ds-0f601751-j488 3812m 12% 6751Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 8123m 27% 16076Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3790m 12% 7460Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8381m 28% 13453Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 8203m 51% 10856Mi 18% 20:34:29 DEBUG --- stderr --- 20:34:29 DEBUG 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING No values available for selected timeframe. 20:35:16 WARNING File /tmp/lodemon_data-20240419_202919/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [19/Apr/2024 20:35:16] "GET /monitoring/average?start_time=20240419_193230&stop_time=20240419_193315 HTTP/1.1" 200 - 20:35:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555328" 20:35:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555328" 20:35:28 INFO [http_cmd]: http status code OK 20:35:28 INFO [http_cmd]: http status code OK 20:35:28 DEBUG --- status code --- 20:35:28 DEBUG --- status code --- 20:35:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:28 DEBUG --- http response --- 20:35:28 DEBUG --- http response --- 20:35:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:28 DEBUG ----- output ----- 20:35:28 DEBUG { 20:35:28 DEBUG "status": "success", 20:35:28 DEBUG "data": { 20:35:28 DEBUG "resultType": "vector", 20:35:28 DEBUG "result": [ 20:35:28 DEBUG { 20:35:28 DEBUG "metric": { 20:35:28 DEBUG "instance": "10.142.0.107:9100" 20:35:28 DEBUG }, 20:35:28 DEBUG "value": [ 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG [...] 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG "instance": "10.142.0.43:9100" 20:35:28 DEBUG }, 20:35:28 DEBUG "value": [ 20:35:28 DEBUG 1713555328, 20:35:28 DEBUG "0.031233333333315728" 20:35:28 DEBUG ] 20:35:28 DEBUG } 20:35:28 DEBUG ] 20:35:28 DEBUG } 20:35:28 DEBUG } 20:35:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:28 DEBUG ----- output ----- 20:35:28 DEBUG { 20:35:28 DEBUG "status": "success", 20:35:28 DEBUG "data": { 20:35:28 DEBUG "resultType": "vector", 20:35:28 DEBUG "result": [ 20:35:28 DEBUG { 20:35:28 DEBUG "metric": { 20:35:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:35:28 DEBUG }, 20:35:28 DEBUG "value": [ 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG [...] 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:35:28 DEBUG }, 20:35:28 DEBUG "value": [ 20:35:28 DEBUG 1713555328, 20:35:28 DEBUG "588.304230911558" 20:35:28 DEBUG ] 20:35:28 DEBUG } 20:35:28 DEBUG ] 20:35:28 DEBUG } 20:35:28 DEBUG } 20:35:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713555328" 20:35:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555328" 20:35:28 ERROR [http_cmd]: ERROR 20:35:28 ERROR --- status code --- 20:35:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:35:28 ERROR --- http response --- 20:35:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:35:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:35:28 INFO [http_cmd]: http status code OK 20:35:28 DEBUG --- status code --- 20:35:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:28 DEBUG --- http response --- 20:35:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:35:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555328" 20:35:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555328" 20:35:28 INFO 20:35:28 INFO [loop_until]: kubectl --namespace=xlou top pods 20:35:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:35:28 INFO [http_cmd]: http status code OK 20:35:28 INFO [http_cmd]: http status code OK 20:35:28 DEBUG --- status code --- 20:35:28 DEBUG --- status code --- 20:35:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:28 DEBUG --- http response --- 20:35:28 DEBUG --- http response --- 20:35:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:28 DEBUG ----- output ----- 20:35:28 DEBUG ----- output ----- 20:35:28 DEBUG { 20:35:28 DEBUG { 20:35:28 DEBUG "status": "success", 20:35:28 DEBUG "status": "success", 20:35:28 DEBUG "data": { 20:35:28 DEBUG "data": { 20:35:28 DEBUG "resultType": "vector", 20:35:28 DEBUG "resultType": "vector", 20:35:28 DEBUG "result": [ 20:35:28 DEBUG "result": [ 20:35:28 DEBUG { 20:35:28 DEBUG { 20:35:28 DEBUG "metric": { 20:35:28 DEBUG "metric": { 20:35:28 DEBUG "pod": "ds-idrepo-0" 20:35:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:35:28 DEBUG }, 20:35:28 DEBUG }, 20:35:28 DEBUG "value": [ 20:35:28 DEBUG "value": [ 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG [...] 20:35:28 DEBUG [...] 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG "pod": "ds-idrepo-2" 20:35:28 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:35:28 DEBUG }, 20:35:28 DEBUG }, 20:35:28 DEBUG "value": [ 20:35:28 DEBUG "value": [ 20:35:28 DEBUG 1713555328, 20:35:28 DEBUG 1713555328, 20:35:28 DEBUG "0" 20:35:28 DEBUG "2059.064808190453" 20:35:28 DEBUG ] 20:35:28 DEBUG ] 20:35:28 DEBUG } 20:35:28 DEBUG } 20:35:28 DEBUG ] 20:35:28 DEBUG ] 20:35:28 DEBUG } 20:35:28 DEBUG } 20:35:28 DEBUG } 20:35:28 DEBUG } 20:35:28 INFO [loop_until]: OK (rc = 0) 20:35:28 DEBUG --- stdout --- 20:35:28 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 3036m 7978Mi am-767f6b67b5-qcqz2 3043m 8593Mi am-767f6b67b5-vbltz 3548m 7961Mi ds-cts-0 5900m 10978Mi ds-cts-1 3559m 11227Mi ds-cts-2 5252m 10984Mi ds-idrepo-0 1496m 3716Mi ds-idrepo-1 1579m 3560Mi ds-idrepo-2 412m 3495Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1174Mi idm-b85f9d978-wnhxb 3m 1217Mi lodemon-68f967fd98-vkx4v 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 377m 131Mi 20:35:28 DEBUG --- stderr --- 20:35:28 DEBUG 20:35:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555328" 20:35:28 INFO [http_cmd]: http status code OK 20:35:28 DEBUG --- status code --- 20:35:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:28 DEBUG --- http response --- 20:35:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:28 DEBUG ----- output ----- 20:35:28 DEBUG { 20:35:28 DEBUG "status": "success", 20:35:28 DEBUG "data": { 20:35:28 DEBUG "resultType": "vector", 20:35:28 DEBUG "result": [ 20:35:28 DEBUG { 20:35:28 DEBUG "metric": { 20:35:28 DEBUG "instance": "10.142.0.107:9100" 20:35:28 DEBUG }, 20:35:28 DEBUG "value": [ 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG [...] 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG 20:35:28 DEBUG "instance": "10.142.0.43:9100" 20:35:28 DEBUG }, 20:35:28 DEBUG "value": [ 20:35:28 DEBUG 1713555328, 20:35:28 DEBUG "0.02000000000000644" 20:35:28 DEBUG ] 20:35:28 DEBUG } 20:35:28 DEBUG ] 20:35:28 DEBUG } 20:35:28 DEBUG } 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG { 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG "data": { 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG "result": [ 20:35:29 DEBUG { 20:35:29 DEBUG "metric": { 20:35:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:35:29 DEBUG }, 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:35:29 DEBUG }, 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG "0" 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG { 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG "data": { 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG "result": [ 20:35:29 DEBUG { 20:35:29 DEBUG "metric": { 20:35:29 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:35:29 DEBUG }, 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:35:29 DEBUG }, 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG "8675344384" 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555329" 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 INFO 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 DEBUG --- status code --- 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG --- status code --- 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 INFO [loop_until]: kubectl --namespace=xlou top node 20:35:29 DEBUG --- status code --- 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG --- status code --- 20:35:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 INFO [http_cmd]: http status code OK 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG --- status code --- 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:35:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG { 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:35:29 DEBUG { 20:35:29 DEBUG { 20:35:29 DEBUG { 20:35:29 DEBUG { 20:35:29 DEBUG { 20:35:29 DEBUG { 20:35:29 DEBUG { 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:35:29 DEBUG { 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG --- http response --- 20:35:29 DEBUG { 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG { 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG { 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:29 DEBUG { 20:35:29 DEBUG { 20:35:29 DEBUG "data": { 20:35:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:35:29 DEBUG "data": { 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG { 20:35:29 DEBUG "data": { 20:35:29 DEBUG "data": { 20:35:29 DEBUG "data": { 20:35:29 DEBUG "data": { 20:35:29 DEBUG "data": { 20:35:29 DEBUG "data": { 20:35:29 DEBUG "data": { 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG { 20:35:29 DEBUG "data": { 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG ----- output ----- 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG "data": { 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG "data": { 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG { 20:35:29 DEBUG "data": { 20:35:29 DEBUG "data": { 20:35:29 DEBUG "result": [ 20:35:29 DEBUG { 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG "data": { 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG "data": { 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG { 20:35:29 DEBUG "status": "success", 20:35:29 DEBUG { 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG { 20:35:29 DEBUG { 20:35:29 DEBUG { 20:35:29 DEBUG { 20:35:29 DEBUG "metric": { 20:35:29 DEBUG { 20:35:29 DEBUG { 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG { 20:35:29 DEBUG "data": { 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "metric": { 20:35:29 DEBUG "data": { 20:35:29 DEBUG "metric": { 20:35:29 DEBUG { 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "metric": { 20:35:29 DEBUG "metric": { 20:35:29 DEBUG "metric": { 20:35:29 DEBUG "pod": "ds-idrepo-0" 20:35:29 DEBUG { 20:35:29 DEBUG "metric": { 20:35:29 DEBUG "metric": { 20:35:29 DEBUG { 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "metric": { 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG { 20:35:29 DEBUG { 20:35:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:35:29 DEBUG "resultType": "vector", 20:35:29 DEBUG "pod": "ds-cts-0" 20:35:29 DEBUG "metric": { 20:35:29 DEBUG { 20:35:29 DEBUG "pod": "ds-idrepo-0" 20:35:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:35:29 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:35:29 DEBUG }, 20:35:29 DEBUG "value": [ 20:35:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:35:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:35:29 DEBUG "metric": { 20:35:29 DEBUG { 20:35:29 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:35:29 DEBUG "result": [ 20:35:29 DEBUG "metric": { 20:35:29 DEBUG "metric": { 20:35:29 DEBUG }, 20:35:29 DEBUG "result": [ 20:35:29 DEBUG }, 20:35:29 DEBUG "instance": "10.142.0.107:9100" 20:35:29 DEBUG "metric": { 20:35:29 DEBUG }, 20:35:29 DEBUG }, 20:35:29 DEBUG }, 20:35:29 DEBUG 20:35:29 DEBUG "metric": { 20:35:29 DEBUG }, 20:35:29 DEBUG }, 20:35:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:35:29 DEBUG "metric": { 20:35:29 DEBUG }, 20:35:29 DEBUG { 20:35:29 DEBUG "metric": { 20:35:29 DEBUG "instance": "10.142.0.107:9100" 20:35:29 DEBUG "value": [ 20:35:29 DEBUG { 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:35:29 DEBUG "value": [ 20:35:29 DEBUG "value": [ 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG "instance": "10.142.0.107:9100" 20:35:29 DEBUG "value": [ 20:35:29 DEBUG }, 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG "pod": "ds-cts-0" 20:35:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:35:29 DEBUG }, 20:35:29 DEBUG 20:35:29 DEBUG "metric": { 20:35:29 DEBUG }, 20:35:29 DEBUG 20:35:29 DEBUG }, 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG }, 20:35:29 DEBUG 20:35:29 DEBUG "pod": "ds-cts-0" 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG }, 20:35:29 DEBUG }, 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG "pod": "ds-idrepo-0" 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG }, 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG "value": [ 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG }, 20:35:29 DEBUG 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG "pod": "ds-idrepo-2" 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG "pod": "ds-cts-2" 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG }, 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG }, 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG }, 20:35:29 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:35:29 DEBUG [...] 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG "pod": "ds-cts-2" 20:35:29 DEBUG "pod": "am-767f6b67b5-qcqz2" 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG "0" 20:35:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG "value": [ 20:35:29 DEBUG }, 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG }, 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:35:29 DEBUG "0" 20:35:29 DEBUG }, 20:35:29 DEBUG }, 20:35:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:35:29 DEBUG 20:35:29 DEBUG ] 20:35:29 DEBUG }, 20:35:29 DEBUG 20:35:29 DEBUG [...] 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG "instance": "10.142.0.43:9100" 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG "instance": "10.142.0.43:9100" 20:35:29 DEBUG 20:35:29 DEBUG }, 20:35:29 DEBUG ] 20:35:29 DEBUG "value": [ 20:35:29 DEBUG "value": [ 20:35:29 DEBUG }, 20:35:29 DEBUG 20:35:29 DEBUG } 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 20:35:29 DEBUG 20:35:29 DEBUG "1378.0666666666666" 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG "pod": "ds-cts-2" 20:35:29 DEBUG }, 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG 20:35:29 DEBUG }, 20:35:29 DEBUG "value": [ 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG } 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG "value": [ 20:35:29 DEBUG "instance": "10.142.0.43:9100" 20:35:29 DEBUG ] 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG "pod": "ds-cts-2" 20:35:29 DEBUG 20:35:29 DEBUG ] 20:35:29 DEBUG "0" 20:35:29 DEBUG }, 20:35:29 DEBUG "value": [ 20:35:29 DEBUG "216470" 20:35:29 DEBUG "pod": "ds-idrepo-2" 20:35:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG "161.86666666666667" 20:35:29 DEBUG ] 20:35:29 DEBUG "4.617476807544807" 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG }, 20:35:29 DEBUG } 20:35:29 DEBUG "161.63333333333333" 20:35:29 DEBUG }, 20:35:29 DEBUG 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG "value": [ 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG ] 20:35:29 DEBUG }, 20:35:29 DEBUG }, 20:35:29 DEBUG "2069750760448" 20:35:29 DEBUG ] 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG "0.0215" 20:35:29 DEBUG "value": [ 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG "value": [ 20:35:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG "26357560.4" 20:35:29 DEBUG } 20:35:29 DEBUG "value": [ 20:35:29 DEBUG "value": [ 20:35:29 DEBUG } 20:35:29 DEBUG "0" 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG } 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG }, 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG "0.00011607777777777776" 20:35:29 DEBUG ] 20:35:29 DEBUG ] 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG ] 20:35:29 DEBUG ] 20:35:29 DEBUG ] 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG "26356338.333333332" 20:35:29 DEBUG ] 20:35:29 DEBUG "0" 20:35:29 DEBUG "value": [ 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG "0" 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG 1713555329, 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG "162.43333333333334" 20:35:29 DEBUG ] 20:35:29 DEBUG "8.023695400354303" 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG ] 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG ] 20:35:29 DEBUG ] 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 DEBUG } 20:35:29 INFO [loop_until]: OK (rc = 0) 20:35:29 DEBUG --- stdout --- 20:35:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 6178Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 650m 2% 7315Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2013m 6% 10710Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3369Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 3964m 13% 13963Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 167m 0% 5175Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2710m 9% 11489Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 1556m 5% 3425Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 2974m 9% 10213Mi 9% gke-xlou-cdm-ds-0f601751-j488 921m 3% 6740Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 3521m 11% 16065Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 890m 2% 7413Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 4258m 14% 13447Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 1582m 9% 3991Mi 6% 20:35:29 DEBUG --- stderr --- 20:35:29 DEBUG 20:36:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555388" 20:36:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555388" 20:36:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713555388" 20:36:28 INFO [http_cmd]: http status code OK 20:36:28 DEBUG --- status code --- 20:36:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:28 DEBUG --- http response --- 20:36:28 INFO [http_cmd]: http status code OK 20:36:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:28 DEBUG --- status code --- 20:36:28 DEBUG ----- output ----- 20:36:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:28 DEBUG { 20:36:28 DEBUG --- http response --- 20:36:28 DEBUG "status": "success", 20:36:28 DEBUG "data": { 20:36:28 DEBUG "resultType": "vector", 20:36:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:36:28 DEBUG "result": [ 20:36:28 DEBUG ----- output ----- 20:36:28 DEBUG { 20:36:28 DEBUG { 20:36:28 DEBUG "metric": { 20:36:28 DEBUG "status": "success", 20:36:28 DEBUG "instance": "10.142.0.107:9100" 20:36:28 DEBUG "data": { 20:36:28 DEBUG }, 20:36:28 DEBUG "resultType": "vector", 20:36:28 DEBUG "value": [ 20:36:28 DEBUG "result": [ 20:36:28 DEBUG 20:36:28 DEBUG { 20:36:28 DEBUG 20:36:28 DEBUG "metric": { 20:36:28 DEBUG 20:36:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:36:28 DEBUG [...] 20:36:28 DEBUG }, 20:36:28 DEBUG 20:36:28 DEBUG "value": [ 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG "instance": "10.142.0.43:9100" 20:36:28 DEBUG 20:36:28 DEBUG }, 20:36:28 DEBUG [...] 20:36:28 DEBUG "value": [ 20:36:28 DEBUG 20:36:28 DEBUG 1713555388, 20:36:28 DEBUG 20:36:28 DEBUG "0.1527333333332564" 20:36:28 DEBUG 20:36:28 DEBUG ] 20:36:28 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:36:28 DEBUG } 20:36:28 DEBUG }, 20:36:28 DEBUG ] 20:36:28 DEBUG "value": [ 20:36:28 DEBUG } 20:36:28 DEBUG 1713555388, 20:36:28 DEBUG } 20:36:28 DEBUG "97265.0922031705" 20:36:28 DEBUG ] 20:36:28 DEBUG } 20:36:28 DEBUG ] 20:36:28 DEBUG } 20:36:28 DEBUG } 20:36:28 ERROR [http_cmd]: ERROR 20:36:28 ERROR --- status code --- 20:36:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:36:28 ERROR --- http response --- 20:36:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:36:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:36:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555388" 20:36:28 INFO [http_cmd]: http status code OK 20:36:28 DEBUG --- status code --- 20:36:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:28 DEBUG --- http response --- 20:36:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:36:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555388" 20:36:28 INFO [http_cmd]: http status code OK 20:36:28 DEBUG --- status code --- 20:36:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:28 DEBUG --- http response --- 20:36:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:36:28 DEBUG ----- output ----- 20:36:28 DEBUG { 20:36:28 DEBUG "status": "success", 20:36:28 DEBUG "data": { 20:36:28 DEBUG "resultType": "vector", 20:36:28 DEBUG "result": [ 20:36:28 DEBUG { 20:36:28 DEBUG "metric": { 20:36:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:36:28 DEBUG }, 20:36:28 DEBUG "value": [ 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG [...] 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:36:28 DEBUG }, 20:36:28 DEBUG "value": [ 20:36:28 DEBUG 1713555388, 20:36:28 DEBUG "1422800.1294079586" 20:36:28 DEBUG ] 20:36:28 DEBUG } 20:36:28 DEBUG ] 20:36:28 DEBUG } 20:36:28 DEBUG } 20:36:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555388" 20:36:28 INFO [http_cmd]: http status code OK 20:36:28 DEBUG --- status code --- 20:36:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:28 DEBUG --- http response --- 20:36:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:28 DEBUG ----- output ----- 20:36:28 DEBUG { 20:36:28 DEBUG "status": "success", 20:36:28 DEBUG "data": { 20:36:28 DEBUG "resultType": "vector", 20:36:28 DEBUG "result": [ 20:36:28 DEBUG { 20:36:28 DEBUG "metric": { 20:36:28 DEBUG "pod": "ds-idrepo-0" 20:36:28 DEBUG }, 20:36:28 DEBUG "value": [ 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG [...] 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG "pod": "ds-idrepo-2" 20:36:28 DEBUG }, 20:36:28 DEBUG "value": [ 20:36:28 DEBUG 1713555388, 20:36:28 DEBUG "0" 20:36:28 DEBUG ] 20:36:28 DEBUG } 20:36:28 DEBUG ] 20:36:28 DEBUG } 20:36:28 DEBUG } 20:36:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555388" 20:36:28 INFO 20:36:28 INFO [loop_until]: kubectl --namespace=xlou top pods 20:36:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:28 INFO [http_cmd]: http status code OK 20:36:28 DEBUG --- status code --- 20:36:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:28 DEBUG --- http response --- 20:36:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:28 DEBUG ----- output ----- 20:36:28 DEBUG { 20:36:28 DEBUG "status": "success", 20:36:28 DEBUG "data": { 20:36:28 DEBUG "resultType": "vector", 20:36:28 DEBUG "result": [ 20:36:28 DEBUG { 20:36:28 DEBUG "metric": { 20:36:28 DEBUG "instance": "10.142.0.107:9100" 20:36:28 DEBUG }, 20:36:28 DEBUG "value": [ 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG [...] 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG 20:36:28 DEBUG "instance": "10.142.0.43:9100" 20:36:28 DEBUG }, 20:36:28 DEBUG "value": [ 20:36:28 DEBUG 1713555388, 20:36:28 DEBUG "0.08600000000000325" 20:36:28 DEBUG ] 20:36:28 DEBUG } 20:36:28 DEBUG ] 20:36:28 DEBUG } 20:36:28 DEBUG } 20:36:29 INFO [loop_until]: OK (rc = 0) 20:36:29 DEBUG --- stdout --- 20:36:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 10975m 7978Mi am-767f6b67b5-qcqz2 10298m 8593Mi am-767f6b67b5-vbltz 13004m 7969Mi ds-cts-0 9263m 10976Mi ds-cts-1 8909m 11226Mi ds-cts-2 9885m 10985Mi ds-idrepo-0 4712m 3716Mi ds-idrepo-1 5102m 3561Mi ds-idrepo-2 6399m 3495Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1174Mi idm-b85f9d978-wnhxb 4m 1217Mi lodemon-68f967fd98-vkx4v 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7600m 2121Mi 20:36:29 DEBUG --- stderr --- 20:36:29 DEBUG 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713555389" 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG { 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "data": { 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "result": [ 20:36:29 DEBUG { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:36:29 DEBUG }, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG [...] 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:36:29 DEBUG }, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG "0" 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG { 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "data": { 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "result": [ 20:36:29 DEBUG { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:36:29 DEBUG }, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG [...] 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:36:29 DEBUG }, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG "557944832" 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG { 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "data": { 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "result": [ 20:36:29 DEBUG { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "pod": "ds-idrepo-0" 20:36:29 DEBUG }, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG [...] 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG "pod": "ds-idrepo-2" 20:36:29 DEBUG }, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 1713555389, 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 DEBUG "0" 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 INFO 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 INFO [loop_until]: kubectl --namespace=xlou top node 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555389" 20:36:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555389" 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555389" 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555389" 20:36:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713555389" 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 DEBUG --- status code --- 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG --- status code --- 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 DEBUG --- status code --- 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:29 DEBUG ----- output ----- 20:36:29 INFO [http_cmd]: http status code OK 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG { 20:36:29 DEBUG --- status code --- 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG { 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG { 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG { 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG { 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "data": { 20:36:29 DEBUG --- http response --- 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG { 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG { 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "data": { 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "data": { 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG { 20:36:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "data": { 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "data": { 20:36:29 DEBUG "data": { 20:36:29 DEBUG "data": { 20:36:29 DEBUG "data": { 20:36:29 DEBUG "data": { 20:36:29 DEBUG "data": { 20:36:29 DEBUG "data": { 20:36:29 DEBUG "data": { 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "data": { 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "result": [ 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG ----- output ----- 20:36:29 DEBUG "data": { 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "data": { 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "result": [ 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "result": [ 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "result": [ 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "result": [ 20:36:29 DEBUG "result": [ 20:36:29 DEBUG "result": [ 20:36:29 DEBUG "result": [ 20:36:29 DEBUG "result": [ 20:36:29 DEBUG "result": [ 20:36:29 DEBUG "result": [ 20:36:29 DEBUG "result": [ 20:36:29 DEBUG { 20:36:29 DEBUG "result": [ 20:36:29 DEBUG "data": { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "status": "success", 20:36:29 DEBUG "result": [ 20:36:29 DEBUG { 20:36:29 DEBUG "result": [ 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG { 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:36:29 DEBUG "instance": "10.142.0.107:9100" 20:36:29 DEBUG "data": { 20:36:29 DEBUG { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "pod": "ds-idrepo-0" 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "result": [ 20:36:29 DEBUG }, 20:36:29 DEBUG }, 20:36:29 DEBUG "resultType": "vector", 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "instance": "10.142.0.107:9100" 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "pod": "ds-cts-0" 20:36:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:36:29 DEBUG "instance": "10.142.0.107:9100" 20:36:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:36:29 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:36:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:36:29 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:36:29 DEBUG "pod": "ds-cts-0" 20:36:29 DEBUG }, 20:36:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:36:29 DEBUG { 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "result": [ 20:36:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:36:29 DEBUG }, 20:36:29 DEBUG "pod": "ds-idrepo-0" 20:36:29 DEBUG }, 20:36:29 DEBUG }, 20:36:29 DEBUG }, 20:36:29 DEBUG }, 20:36:29 DEBUG }, 20:36:29 DEBUG }, 20:36:29 DEBUG }, 20:36:29 DEBUG }, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG }, 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "pod": "ds-cts-0" 20:36:29 DEBUG 20:36:29 DEBUG { 20:36:29 DEBUG }, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG }, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 20:36:29 DEBUG }, 20:36:29 DEBUG 20:36:29 DEBUG "metric": { 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 20:36:29 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG [...] 20:36:29 DEBUG }, 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG [...] 20:36:29 DEBUG 20:36:29 DEBUG [...] 20:36:29 DEBUG 20:36:29 DEBUG [...] 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 20:36:29 DEBUG [...] 20:36:29 DEBUG 20:36:29 DEBUG [...] 20:36:29 DEBUG [...] 20:36:29 DEBUG [...] 20:36:29 DEBUG [...] 20:36:29 DEBUG [...] 20:36:29 DEBUG [...] 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG [...] 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG [...] 20:36:29 DEBUG 20:36:29 DEBUG [...] 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG [...] 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG [...] 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG "instance": "10.142.0.43:9100" 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:36:29 DEBUG }, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG "0.000028682909394188697" 20:36:29 DEBUG ] 20:36:29 DEBUG }, 20:36:29 DEBUG [...] 20:36:29 DEBUG 20:36:29 DEBUG "instance": "10.142.0.43:9100" 20:36:29 DEBUG 20:36:29 DEBUG "pod": "ds-cts-2" 20:36:29 DEBUG }, 20:36:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:36:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:36:29 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:36:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:36:29 DEBUG 20:36:29 DEBUG "pod": "ds-cts-2" 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG 20:36:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 20:36:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:36:29 DEBUG }, 20:36:29 DEBUG "pod": "ds-idrepo-2" 20:36:29 DEBUG "instance": "10.142.0.43:9100" 20:36:29 DEBUG "value": [ 20:36:29 DEBUG }, 20:36:29 DEBUG }, 20:36:29 DEBUG }, 20:36:29 DEBUG }, 20:36:29 DEBUG 20:36:29 DEBUG }, 20:36:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:36:29 DEBUG } 20:36:29 DEBUG "pod": "ds-cts-2" 20:36:29 DEBUG }, 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG 20:36:29 DEBUG }, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG }, 20:36:29 DEBUG }, 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "pod": "ds-cts-2" 20:36:29 DEBUG "value": [ 20:36:29 DEBUG }, 20:36:29 DEBUG ] 20:36:29 DEBUG }, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "45758278.86666666" 20:36:29 DEBUG 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "0" 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG }, 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG "value": [ 20:36:29 DEBUG } 20:36:29 DEBUG "value": [ 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG ] 20:36:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG "45757586.400000006" 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG "1289.3666666666666" 20:36:29 DEBUG "0" 20:36:29 DEBUG "1899.6666666666667" 20:36:29 DEBUG "value": [ 20:36:29 DEBUG "0" 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG } 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG "0.022" 20:36:29 DEBUG } 20:36:29 DEBUG }, 20:36:29 DEBUG "1899" 20:36:29 DEBUG ] 20:36:29 DEBUG "0" 20:36:29 DEBUG "2070827887616" 20:36:29 DEBUG "178009" 20:36:29 DEBUG ] 20:36:29 DEBUG ] 20:36:29 DEBUG ] 20:36:29 DEBUG ] 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG ] 20:36:29 DEBUG "1897.2" 20:36:29 DEBUG "0.00003333333333333336" 20:36:29 DEBUG ] 20:36:29 DEBUG ] 20:36:29 DEBUG "value": [ 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG ] 20:36:29 DEBUG ] 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG "0" 20:36:29 DEBUG } 20:36:29 DEBUG ] 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG 1713555389, 20:36:29 DEBUG } 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG ] 20:36:29 DEBUG ] 20:36:29 DEBUG ] 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG "3.6318923155227627" 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG ] 20:36:29 DEBUG ] 20:36:29 DEBUG ] 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG ] 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG ] 20:36:29 DEBUG } 20:36:29 INFO [loop_until]: OK (rc = 0) 20:36:29 DEBUG } 20:36:29 DEBUG } 20:36:29 DEBUG --- stdout --- 20:36:29 DEBUG } 20:36:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 59m 0% 6172Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4974m 16% 7320Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11165m 37% 10684Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3377Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 9772m 32% 13969Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 5186Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11358m 38% 11476Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 11742m 39% 3656Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 12654m 42% 10192Mi 9% gke-xlou-cdm-ds-0f601751-j488 5458m 18% 6740Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 8978m 30% 16064Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5446m 18% 7407Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9953m 33% 13444Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7602m 47% 5802Mi 9% 20:36:29 DEBUG --- stderr --- 20:36:29 DEBUG 20:37:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555448" 20:37:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555448" 20:37:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713555448" 20:37:28 INFO [http_cmd]: http status code OK 20:37:28 DEBUG --- status code --- 20:37:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:28 DEBUG --- http response --- 20:37:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:28 DEBUG ----- output ----- 20:37:28 DEBUG { 20:37:28 DEBUG "status": "success", 20:37:28 DEBUG "data": { 20:37:28 DEBUG "resultType": "vector", 20:37:28 DEBUG "result": [ 20:37:28 DEBUG { 20:37:28 DEBUG "metric": { 20:37:28 DEBUG "instance": "10.142.0.107:9100" 20:37:28 DEBUG }, 20:37:28 DEBUG "value": [ 20:37:28 DEBUG 20:37:28 DEBUG 20:37:28 DEBUG 20:37:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555448" 20:37:28 DEBUG [...] 20:37:28 ERROR [http_cmd]: ERROR 20:37:28 DEBUG 20:37:28 INFO [http_cmd]: http status code OK 20:37:28 ERROR --- status code --- 20:37:28 DEBUG 20:37:28 DEBUG --- status code --- 20:37:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:37:28 DEBUG 20:37:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:28 ERROR --- http response --- 20:37:28 DEBUG "instance": "10.142.0.43:9100" 20:37:28 DEBUG --- http response --- 20:37:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:37:28 DEBUG }, 20:37:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:37:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:37:28 DEBUG "value": [ 20:37:28 DEBUG ----- output ----- 20:37:28 DEBUG 1713555448, 20:37:28 DEBUG { 20:37:28 DEBUG "0.1519000000000498" 20:37:28 DEBUG "status": "success", 20:37:28 DEBUG ] 20:37:28 DEBUG "data": { 20:37:28 DEBUG } 20:37:28 DEBUG "resultType": "vector", 20:37:28 DEBUG ] 20:37:28 DEBUG "result": [ 20:37:28 DEBUG } 20:37:28 DEBUG { 20:37:28 DEBUG } 20:37:28 DEBUG "metric": { 20:37:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:37:28 DEBUG }, 20:37:28 DEBUG "value": [ 20:37:28 DEBUG 20:37:28 DEBUG 20:37:28 DEBUG 20:37:28 DEBUG [...] 20:37:28 DEBUG 20:37:28 DEBUG 20:37:28 DEBUG 20:37:28 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:37:28 DEBUG }, 20:37:28 DEBUG "value": [ 20:37:28 DEBUG 1713555448, 20:37:28 DEBUG "0" 20:37:28 DEBUG ] 20:37:28 DEBUG } 20:37:28 DEBUG ] 20:37:28 DEBUG } 20:37:28 DEBUG } 20:37:28 INFO [http_cmd]: http status code OK 20:37:28 DEBUG --- status code --- 20:37:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:28 DEBUG --- http response --- 20:37:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:37:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555448" 20:37:28 INFO [http_cmd]: http status code OK 20:37:28 DEBUG --- status code --- 20:37:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:28 DEBUG --- http response --- 20:37:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:37:28 DEBUG ----- output ----- 20:37:28 DEBUG { 20:37:28 DEBUG "status": "success", 20:37:28 DEBUG "data": { 20:37:28 DEBUG "resultType": "vector", 20:37:28 DEBUG "result": [ 20:37:28 DEBUG { 20:37:28 DEBUG "metric": { 20:37:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:37:28 DEBUG }, 20:37:28 DEBUG "value": [ 20:37:28 DEBUG 20:37:28 DEBUG 20:37:28 DEBUG 20:37:28 DEBUG [...] 20:37:28 DEBUG 20:37:28 DEBUG 20:37:28 DEBUG 20:37:28 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:37:28 DEBUG }, 20:37:28 DEBUG "value": [ 20:37:28 DEBUG 1713555448, 20:37:28 DEBUG "41699713.897339635" 20:37:28 DEBUG ] 20:37:28 DEBUG } 20:37:28 DEBUG ] 20:37:28 DEBUG } 20:37:28 DEBUG } 20:37:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555448" 20:37:28 INFO [http_cmd]: http status code OK 20:37:28 DEBUG --- status code --- 20:37:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:28 DEBUG --- http response --- 20:37:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:28 DEBUG ----- output ----- 20:37:28 DEBUG { 20:37:28 DEBUG "status": "success", 20:37:28 DEBUG "data": { 20:37:28 DEBUG "resultType": "vector", 20:37:28 DEBUG "result": [ 20:37:28 DEBUG { 20:37:28 DEBUG "metric": { 20:37:28 DEBUG "pod": "ds-idrepo-0" 20:37:28 DEBUG }, 20:37:28 DEBUG "value": [ 20:37:28 DEBUG 20:37:28 DEBUG 20:37:28 DEBUG 20:37:28 DEBUG [...] 20:37:28 DEBUG 20:37:28 DEBUG 20:37:28 DEBUG 20:37:28 DEBUG "pod": "ds-idrepo-2" 20:37:28 DEBUG }, 20:37:28 DEBUG "value": [ 20:37:28 DEBUG 1713555448, 20:37:28 DEBUG "0" 20:37:28 DEBUG ] 20:37:28 DEBUG } 20:37:28 DEBUG ] 20:37:28 DEBUG } 20:37:28 DEBUG } 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555449" 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG { 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG "data": { 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "result": [ 20:37:29 DEBUG { 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "instance": "10.142.0.107:9100" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG "instance": "10.142.0.43:9100" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG "0.08866666666667126" 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 INFO 20:37:29 INFO [loop_until]: kubectl --namespace=xlou top pods 20:37:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:37:29 INFO [loop_until]: OK (rc = 0) 20:37:29 DEBUG --- stdout --- 20:37:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 8594m 8986Mi am-767f6b67b5-qcqz2 8639m 8597Mi am-767f6b67b5-vbltz 9341m 8975Mi ds-cts-0 7873m 10977Mi ds-cts-1 7051m 11242Mi ds-cts-2 7968m 10999Mi ds-idrepo-0 3684m 3715Mi ds-idrepo-1 3662m 3561Mi ds-idrepo-2 3754m 3495Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1174Mi idm-b85f9d978-wnhxb 3m 1217Mi lodemon-68f967fd98-vkx4v 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8033m 5398Mi 20:37:29 DEBUG --- stderr --- 20:37:29 DEBUG 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713555449" 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG { 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG "data": { 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "result": [ 20:37:29 DEBUG { 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG "0" 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG { 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG "data": { 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "result": [ 20:37:29 DEBUG { 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG "4215914496" 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG { 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG "data": { 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "result": [ 20:37:29 DEBUG { 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "pod": "ds-idrepo-0" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG "pod": "ds-idrepo-2" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG "0" 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG { 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG "data": { 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "result": [ 20:37:29 DEBUG { 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG "7.99968529539103" 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 INFO 20:37:29 INFO [loop_until]: kubectl --namespace=xlou top node 20:37:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG { 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG "data": { 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "result": [ 20:37:29 DEBUG { 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG "1246.1666666666667" 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG { 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG "data": { 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "result": [ 20:37:29 DEBUG { 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG "1353.9666666666667" 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555449" 20:37:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713555449" 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG --- status code --- 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG --- status code --- 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG --- status code --- 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG --- status code --- 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:29 INFO [http_cmd]: http status code OK 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:29 DEBUG ----- output ----- 20:37:29 INFO [loop_until]: OK (rc = 0) 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG --- status code --- 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:37:29 DEBUG { 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG { 20:37:29 DEBUG --- stdout --- 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:29 DEBUG { 20:37:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:37:29 DEBUG { 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG { 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG { 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6179Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3817m 12% 7324Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8579m 28% 11681Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3378Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 7987m 26% 13957Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5179Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8647m 28% 11471Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 8340m 27% 3684Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 9933m 33% 11173Mi 10% gke-xlou-cdm-ds-0f601751-j488 3944m 13% 6744Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 7257m 24% 16067Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3817m 12% 7398Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 7762m 25% 13439Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 8206m 51% 8915Mi 15% 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG --- http response --- 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG { 20:37:29 DEBUG { 20:37:29 DEBUG { 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG { 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:37:29 DEBUG "data": { 20:37:29 DEBUG { 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG "data": { 20:37:29 DEBUG { 20:37:29 DEBUG --- stderr --- 20:37:29 DEBUG "data": { 20:37:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:37:29 DEBUG "data": { 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG "data": { 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG "data": { 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG ----- output ----- 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "data": { 20:37:29 DEBUG "data": { 20:37:29 DEBUG "data": { 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "data": { 20:37:29 DEBUG { 20:37:29 DEBUG "result": [ 20:37:29 DEBUG "data": { 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "result": [ 20:37:29 DEBUG "data": { 20:37:29 DEBUG "result": [ 20:37:29 DEBUG { 20:37:29 DEBUG "result": [ 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "result": [ 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG { 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "result": [ 20:37:29 DEBUG { 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG { 20:37:29 DEBUG "status": "success", 20:37:29 DEBUG { 20:37:29 DEBUG "result": [ 20:37:29 DEBUG "result": [ 20:37:29 DEBUG "result": [ 20:37:29 DEBUG { 20:37:29 DEBUG "result": [ 20:37:29 DEBUG "data": { 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "result": [ 20:37:29 DEBUG { 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "result": [ 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "data": { 20:37:29 DEBUG "metric": { 20:37:29 DEBUG { 20:37:29 DEBUG { 20:37:29 DEBUG { 20:37:29 DEBUG "metric": { 20:37:29 DEBUG { 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:37:29 DEBUG { 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:37:29 DEBUG { 20:37:29 DEBUG "instance": "10.142.0.107:9100" 20:37:29 DEBUG "resultType": "vector", 20:37:29 DEBUG "pod": "ds-cts-0" 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "result": [ 20:37:29 DEBUG }, 20:37:29 DEBUG "metric": { 20:37:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:37:29 DEBUG }, 20:37:29 DEBUG "metric": { 20:37:29 DEBUG }, 20:37:29 DEBUG "result": [ 20:37:29 DEBUG }, 20:37:29 DEBUG "pod": "ds-cts-0" 20:37:29 DEBUG "instance": "10.142.0.107:9100" 20:37:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:37:29 DEBUG }, 20:37:29 DEBUG "pod": "ds-idrepo-0" 20:37:29 DEBUG { 20:37:29 DEBUG "value": [ 20:37:29 DEBUG "pod": "ds-idrepo-0" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG "instance": "10.142.0.107:9100" 20:37:29 DEBUG "value": [ 20:37:29 DEBUG { 20:37:29 DEBUG "value": [ 20:37:29 DEBUG }, 20:37:29 DEBUG }, 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG }, 20:37:29 DEBUG "metric": { 20:37:29 DEBUG 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 20:37:29 DEBUG }, 20:37:29 DEBUG 20:37:29 DEBUG "metric": { 20:37:29 DEBUG 20:37:29 DEBUG "value": [ 20:37:29 DEBUG "value": [ 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 20:37:29 DEBUG "value": [ 20:37:29 DEBUG "pod": "ds-cts-0" 20:37:29 DEBUG 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 20:37:29 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG }, 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG }, 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG "value": [ 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG "value": [ 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG [...] 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG [...] 20:37:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:37:29 DEBUG 20:37:29 DEBUG "instance": "10.142.0.43:9100" 20:37:29 DEBUG [...] 20:37:29 DEBUG "pod": "ds-cts-2" 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:37:29 DEBUG 20:37:29 DEBUG 20:37:29 DEBUG }, 20:37:29 DEBUG 20:37:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:37:29 DEBUG }, 20:37:29 DEBUG 20:37:29 DEBUG }, 20:37:29 DEBUG 20:37:29 DEBUG }, 20:37:29 DEBUG "pod": "ds-cts-2" 20:37:29 DEBUG "instance": "10.142.0.43:9100" 20:37:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:37:29 DEBUG }, 20:37:29 DEBUG "pod": "ds-cts-2" 20:37:29 DEBUG 20:37:29 DEBUG "value": [ 20:37:29 DEBUG "pod": "ds-idrepo-2" 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG "instance": "10.142.0.43:9100" 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 20:37:29 DEBUG "value": [ 20:37:29 DEBUG }, 20:37:29 DEBUG }, 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG }, 20:37:29 DEBUG 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG }, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG }, 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG "value": [ 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG "value": [ 20:37:29 DEBUG "pod": "ds-cts-2" 20:37:29 DEBUG "1352.6333333333332" 20:37:29 DEBUG "value": [ 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG "0" 20:37:29 DEBUG "value": [ 20:37:29 DEBUG "40377639.233333334" 20:37:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:37:29 DEBUG "0" 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG "0.022" 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG }, 20:37:29 DEBUG ] 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG "190121.5" 20:37:29 DEBUG ] 20:37:29 DEBUG ] 20:37:29 DEBUG ] 20:37:29 DEBUG }, 20:37:29 DEBUG ] 20:37:29 DEBUG "0.000019882222222222222" 20:37:29 DEBUG "2073068663808" 20:37:29 DEBUG "1353.9333333333334" 20:37:29 DEBUG ] 20:37:29 DEBUG "0" 20:37:29 DEBUG "value": [ 20:37:29 DEBUG } 20:37:29 DEBUG "0" 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG "value": [ 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG ] 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG ] 20:37:29 DEBUG ] 20:37:29 DEBUG "40378225.13333333" 20:37:29 DEBUG ] 20:37:29 DEBUG ] 20:37:29 DEBUG ] 20:37:29 DEBUG 1713555449, 20:37:29 DEBUG "8.024894025579936" 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG "0" 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG ] 20:37:29 DEBUG ] 20:37:29 DEBUG ] 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG ] 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:37:29 DEBUG } 20:38:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555508" 20:38:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713555508" 20:38:28 INFO [http_cmd]: http status code OK 20:38:28 DEBUG --- status code --- 20:38:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:28 DEBUG --- http response --- 20:38:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:38:28 DEBUG ----- output ----- 20:38:28 DEBUG { 20:38:28 DEBUG "status": "success", 20:38:28 DEBUG "data": { 20:38:28 DEBUG "resultType": "vector", 20:38:28 DEBUG "result": [ 20:38:28 DEBUG { 20:38:28 DEBUG "metric": { 20:38:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:38:28 DEBUG }, 20:38:28 DEBUG "value": [ 20:38:28 DEBUG 20:38:28 DEBUG 20:38:28 DEBUG 20:38:28 DEBUG [...] 20:38:28 DEBUG 20:38:28 DEBUG 20:38:28 DEBUG 20:38:28 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:38:28 DEBUG }, 20:38:28 DEBUG "value": [ 20:38:28 DEBUG 1713555508, 20:38:28 DEBUG "0" 20:38:28 DEBUG ] 20:38:28 DEBUG } 20:38:28 DEBUG ] 20:38:28 DEBUG } 20:38:28 DEBUG } 20:38:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555508" 20:38:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555508" 20:38:28 ERROR [http_cmd]: ERROR 20:38:28 ERROR --- status code --- 20:38:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:38:28 ERROR --- http response --- 20:38:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:38:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:38:28 INFO [http_cmd]: http status code OK 20:38:28 INFO [http_cmd]: http status code OK 20:38:28 DEBUG --- status code --- 20:38:28 DEBUG --- status code --- 20:38:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:28 DEBUG --- http response --- 20:38:28 DEBUG --- http response --- 20:38:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:38:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:28 DEBUG ----- output ----- 20:38:28 DEBUG { 20:38:28 DEBUG "status": "success", 20:38:28 DEBUG "data": { 20:38:28 DEBUG "resultType": "vector", 20:38:28 DEBUG "result": [ 20:38:28 DEBUG { 20:38:28 DEBUG "metric": { 20:38:28 DEBUG "instance": "10.142.0.107:9100" 20:38:28 DEBUG }, 20:38:28 DEBUG "value": [ 20:38:28 DEBUG 20:38:28 DEBUG 20:38:28 DEBUG 20:38:28 DEBUG [...] 20:38:28 DEBUG 20:38:28 DEBUG 20:38:28 DEBUG 20:38:28 DEBUG "instance": "10.142.0.43:9100" 20:38:28 DEBUG }, 20:38:28 DEBUG "value": [ 20:38:28 DEBUG 1713555508, 20:38:28 DEBUG "0.035266666666728945" 20:38:28 DEBUG ] 20:38:28 DEBUG } 20:38:28 DEBUG ] 20:38:28 DEBUG } 20:38:28 DEBUG } 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG { 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "data": { 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "result": [ 20:38:29 DEBUG { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG "48018108.738969535" 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- http response --- 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG { 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "data": { 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "result": [ 20:38:29 DEBUG { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "instance": "10.142.0.107:9100" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:29 DEBUG [...] 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG 20:38:29 DEBUG { 20:38:29 DEBUG 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG 20:38:29 DEBUG "data": { 20:38:29 DEBUG "instance": "10.142.0.43:9100" 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG }, 20:38:29 DEBUG "result": [ 20:38:29 DEBUG "value": [ 20:38:29 DEBUG { 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "0.02566666666667932" 20:38:29 DEBUG "pod": "ds-idrepo-0" 20:38:29 DEBUG ] 20:38:29 DEBUG }, 20:38:29 DEBUG } 20:38:29 DEBUG "value": [ 20:38:29 DEBUG ] 20:38:29 DEBUG 20:38:29 DEBUG } 20:38:29 DEBUG 20:38:29 DEBUG } 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "pod": "ds-idrepo-2" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG "0" 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 INFO 20:38:29 INFO [loop_until]: kubectl --namespace=xlou top pods 20:38:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:38:29 INFO [loop_until]: OK (rc = 0) 20:38:29 DEBUG --- stdout --- 20:38:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 7034m 8987Mi am-767f6b67b5-qcqz2 7107m 9044Mi am-767f6b67b5-vbltz 7608m 8974Mi ds-cts-0 5370m 11002Mi ds-cts-1 4396m 11253Mi ds-cts-2 4266m 11006Mi ds-idrepo-0 3284m 3715Mi ds-idrepo-1 3186m 3560Mi ds-idrepo-2 3268m 3495Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1174Mi idm-b85f9d978-wnhxb 3m 1217Mi lodemon-68f967fd98-vkx4v 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8269m 7138Mi 20:38:29 DEBUG --- stderr --- 20:38:29 DEBUG 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713555509" 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG { 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "data": { 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "result": [ 20:38:29 DEBUG { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG "0" 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG { 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "data": { 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "result": [ 20:38:29 DEBUG { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG "7159783424" 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG { 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "data": { 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "result": [ 20:38:29 DEBUG { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "pod": "ds-idrepo-0" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "pod": "ds-idrepo-2" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG "0" 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG { 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "data": { 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "result": [ 20:38:29 DEBUG { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG "8.000462928977601" 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG { 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "data": { 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "result": [ 20:38:29 DEBUG { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG "162.6" 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG { 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "data": { 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "result": [ 20:38:29 DEBUG { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG "1317.9" 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 INFO 20:38:29 INFO [loop_until]: kubectl --namespace=xlou top node 20:38:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555509" 20:38:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713555509" 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG --- status code --- 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG --- status code --- 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG --- status code --- 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- status code --- 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:29 DEBUG ----- output ----- 20:38:29 INFO [http_cmd]: http status code OK 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG { 20:38:29 DEBUG --- status code --- 20:38:29 DEBUG { 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG { 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG { 20:38:29 DEBUG { 20:38:29 DEBUG { 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG { 20:38:29 DEBUG { 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:29 DEBUG { 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG { 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "data": { 20:38:29 DEBUG --- http response --- 20:38:29 DEBUG "data": { 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "data": { 20:38:29 DEBUG { 20:38:29 DEBUG { 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "data": { 20:38:29 DEBUG "data": { 20:38:29 DEBUG "data": { 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:38:29 DEBUG "data": { 20:38:29 DEBUG "data": { 20:38:29 DEBUG { 20:38:29 DEBUG "data": { 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "data": { 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "result": [ 20:38:29 DEBUG "result": [ 20:38:29 DEBUG ----- output ----- 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "result": [ 20:38:29 DEBUG "data": { 20:38:29 DEBUG "data": { 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "result": [ 20:38:29 DEBUG "result": [ 20:38:29 DEBUG "result": [ 20:38:29 DEBUG { 20:38:29 DEBUG { 20:38:29 DEBUG { 20:38:29 DEBUG "result": [ 20:38:29 DEBUG "result": [ 20:38:29 DEBUG "data": { 20:38:29 DEBUG "result": [ 20:38:29 DEBUG { 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG "result": [ 20:38:29 DEBUG { 20:38:29 DEBUG { 20:38:29 DEBUG { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "status": "success", 20:38:29 DEBUG { 20:38:29 DEBUG { 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "result": [ 20:38:29 DEBUG "result": [ 20:38:29 DEBUG "metric": { 20:38:29 DEBUG { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:38:29 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:38:29 DEBUG }, 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "result": [ 20:38:29 DEBUG { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG { 20:38:29 DEBUG { 20:38:29 DEBUG "instance": "10.142.0.107:9100" 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:38:29 DEBUG "instance": "10.142.0.107:9100" 20:38:29 DEBUG }, 20:38:29 DEBUG "data": { 20:38:29 DEBUG "value": [ 20:38:29 DEBUG "pod": "ds-cts-0" 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:38:29 DEBUG "metric": { 20:38:29 DEBUG "metric": { 20:38:29 DEBUG }, 20:38:29 DEBUG "pod": "ds-cts-0" 20:38:29 DEBUG }, 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG "resultType": "vector", 20:38:29 DEBUG 20:38:29 DEBUG "pod": "ds-idrepo-0" 20:38:29 DEBUG 20:38:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:38:29 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:38:29 DEBUG }, 20:38:29 DEBUG "pod": "ds-cts-0" 20:38:29 DEBUG "pod": "ds-idrepo-0" 20:38:29 DEBUG "value": [ 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG "result": [ 20:38:29 DEBUG 20:38:29 DEBUG }, 20:38:29 DEBUG 20:38:29 DEBUG }, 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG }, 20:38:29 DEBUG }, 20:38:29 DEBUG 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG { 20:38:29 DEBUG 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG "value": [ 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "metric": { 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG "instance": "10.142.0.107:9100" 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG }, 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG [...] 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG "pod": "ds-cts-2" 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG }, 20:38:29 DEBUG [...] 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG [...] 20:38:29 DEBUG }, 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 INFO [loop_until]: OK (rc = 0) 20:38:29 DEBUG 20:38:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:38:29 DEBUG --- stdout --- 20:38:29 DEBUG "instance": "10.142.0.43:9100" 20:38:29 DEBUG }, 20:38:29 DEBUG 20:38:29 DEBUG "pod": "ds-cts-2" 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG "instance": "10.142.0.43:9100" 20:38:29 DEBUG "pod": "ds-idrepo-2" 20:38:29 DEBUG "pod": "ds-cts-2" 20:38:29 DEBUG }, 20:38:29 DEBUG }, 20:38:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 6175Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3377m 11% 7316Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6660m 22% 11678Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3361Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 3888m 13% 13991Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 163m 0% 5203Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 6989m 23% 11923Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 8305m 27% 3696Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 7787m 26% 11204Mi 10% gke-xlou-cdm-ds-0f601751-j488 3550m 11% 6743Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 3435m 11% 16089Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3491m 11% 7411Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 3982m 13% 13458Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7928m 49% 10597Mi 18% 20:38:29 DEBUG "value": [ 20:38:29 DEBUG [...] 20:38:29 DEBUG }, 20:38:29 DEBUG "0" 20:38:29 DEBUG 20:38:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:38:29 DEBUG 20:38:29 DEBUG }, 20:38:29 DEBUG "0.000039764444444444444" 20:38:29 DEBUG }, 20:38:29 DEBUG }, 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG "value": [ 20:38:29 DEBUG --- stderr --- 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG 20:38:29 DEBUG 20:38:29 DEBUG ] 20:38:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:38:29 DEBUG }, 20:38:29 DEBUG 20:38:29 DEBUG "value": [ 20:38:29 DEBUG ] 20:38:29 DEBUG "value": [ 20:38:29 DEBUG "value": [ 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG 20:38:29 DEBUG "1319.6666666666667" 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 20:38:29 DEBUG } 20:38:29 DEBUG }, 20:38:29 DEBUG "value": [ 20:38:29 DEBUG "pod": "ds-cts-2" 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG } 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG "210736" 20:38:29 DEBUG "37082132.56666666" 20:38:29 DEBUG ] 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG "instance": "10.142.0.43:9100" 20:38:29 DEBUG ] 20:38:29 DEBUG "value": [ 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG }, 20:38:29 DEBUG "8.032186875480132" 20:38:29 DEBUG ] 20:38:29 DEBUG "2075203773440" 20:38:29 DEBUG ] 20:38:29 DEBUG "0" 20:38:29 DEBUG ] 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG "0" 20:38:29 DEBUG }, 20:38:29 DEBUG } 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG "0.022" 20:38:29 DEBUG "value": [ 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG "0" 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG "value": [ 20:38:29 DEBUG } 20:38:29 DEBUG "1318.3333333333333" 20:38:29 DEBUG ] 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG 1713555509, 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG "0" 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG "37083872.2" 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG ] 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG } 20:38:29 DEBUG } 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING No values available for selected timeframe. 20:39:15 WARNING File /tmp/lodemon_data-20240419_202919/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [19/Apr/2024 20:39:15] "GET /monitoring/average?start_time=20240419_193716&stop_time=20240419_193714 HTTP/1.1" 200 - 20:39:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555568" 20:39:28 INFO [http_cmd]: http status code OK 20:39:28 DEBUG --- status code --- 20:39:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:28 DEBUG --- http response --- 20:39:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:39:28 DEBUG ----- output ----- 20:39:28 DEBUG { 20:39:28 DEBUG "status": "success", 20:39:28 DEBUG "data": { 20:39:28 DEBUG "resultType": "vector", 20:39:28 DEBUG "result": [ 20:39:28 DEBUG { 20:39:28 DEBUG "metric": { 20:39:28 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:39:28 DEBUG }, 20:39:28 DEBUG "value": [ 20:39:28 DEBUG 20:39:28 DEBUG 20:39:28 DEBUG 20:39:28 DEBUG [...] 20:39:28 DEBUG 20:39:28 DEBUG 20:39:28 DEBUG 20:39:28 DEBUG "pod": "idm-b85f9d978-txpwv" 20:39:28 DEBUG }, 20:39:28 DEBUG "value": [ 20:39:28 DEBUG 1713555568, 20:39:28 DEBUG "466.68755518842397" 20:39:28 DEBUG ] 20:39:28 DEBUG } 20:39:28 DEBUG ] 20:39:28 DEBUG } 20:39:28 DEBUG } 20:39:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713555568" 20:39:28 ERROR [http_cmd]: ERROR 20:39:28 ERROR --- status code --- 20:39:28 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:39:28 ERROR --- http response --- 20:39:28 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:39:28 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:39:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555568" 20:39:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555568" 20:39:28 INFO [http_cmd]: http status code OK 20:39:28 INFO [http_cmd]: http status code OK 20:39:28 DEBUG --- status code --- 20:39:28 DEBUG --- status code --- 20:39:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:28 DEBUG --- http response --- 20:39:28 DEBUG --- http response --- 20:39:28 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:39:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:28 DEBUG ----- output ----- 20:39:28 DEBUG { 20:39:28 DEBUG "status": "success", 20:39:28 DEBUG "data": { 20:39:28 DEBUG "resultType": "vector", 20:39:28 DEBUG "result": [ 20:39:28 DEBUG { 20:39:28 DEBUG "metric": { 20:39:28 DEBUG "instance": "10.142.0.107:9100" 20:39:28 DEBUG }, 20:39:28 DEBUG "value": [ 20:39:28 DEBUG 20:39:28 DEBUG 20:39:28 DEBUG 20:39:28 DEBUG [...] 20:39:28 DEBUG 20:39:28 DEBUG 20:39:28 DEBUG 20:39:28 DEBUG "instance": "10.142.0.43:9100" 20:39:28 DEBUG }, 20:39:28 DEBUG "value": [ 20:39:28 DEBUG 1713555568, 20:39:28 DEBUG "0.028233333333310875" 20:39:28 DEBUG ] 20:39:28 DEBUG } 20:39:28 DEBUG ] 20:39:28 DEBUG } 20:39:28 DEBUG } 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555569" 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG { 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG "data": { 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG "result": [ 20:39:29 DEBUG { 20:39:29 DEBUG "metric": { 20:39:29 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG [...] 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "pod": "idm-b85f9d978-txpwv" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG "4666.87555188424" 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG { 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG "data": { 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG "result": [ 20:39:29 DEBUG { 20:39:29 DEBUG "metric": { 20:39:29 DEBUG "instance": "10.142.0.107:9100" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG [...] 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "instance": "10.142.0.43:9100" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG "0.02100000000000174" 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG { 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG "data": { 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG "result": [ 20:39:29 DEBUG { 20:39:29 DEBUG "metric": { 20:39:29 DEBUG "pod": "ds-idrepo-0" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG [...] 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "pod": "ds-idrepo-2" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG "0" 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 INFO 20:39:29 INFO [loop_until]: kubectl --namespace=xlou top pods 20:39:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713555569" 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG { 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG "data": { 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG "result": [ 20:39:29 DEBUG { 20:39:29 DEBUG "metric": { 20:39:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG [...] 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG "0" 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG { 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG "data": { 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG "result": [ 20:39:29 DEBUG { 20:39:29 DEBUG "metric": { 20:39:29 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG [...] 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG "8444211200" 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 INFO [loop_until]: OK (rc = 0) 20:39:29 DEBUG --- stdout --- 20:39:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 2617m 8342Mi am-767f6b67b5-qcqz2 2495m 7562Mi am-767f6b67b5-vbltz 2403m 8024Mi ds-cts-0 5883m 10980Mi ds-cts-1 5716m 11247Mi ds-cts-2 5271m 10986Mi ds-idrepo-0 1304m 3715Mi ds-idrepo-1 624m 3561Mi ds-idrepo-2 962m 3495Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1174Mi idm-b85f9d978-wnhxb 3m 1217Mi lodemon-68f967fd98-vkx4v 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1657m 16Mi 20:39:29 DEBUG --- stderr --- 20:39:29 DEBUG 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG { 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG "data": { 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG "result": [ 20:39:29 DEBUG { 20:39:29 DEBUG "metric": { 20:39:29 DEBUG "pod": "ds-idrepo-0" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG [...] 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "pod": "ds-idrepo-2" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG "0" 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG { 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG "data": { 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG "result": [ 20:39:29 DEBUG { 20:39:29 DEBUG "metric": { 20:39:29 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG [...] 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "pod": "idm-b85f9d978-txpwv" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG "0.0032401515367299046" 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG { 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG "data": { 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG "result": [ 20:39:29 DEBUG { 20:39:29 DEBUG "metric": { 20:39:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG [...] 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG "1896.1666666666667" 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG { 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG "data": { 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG "result": [ 20:39:29 DEBUG { 20:39:29 DEBUG "metric": { 20:39:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG [...] 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG "85.26666666666667" 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG { 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG "data": { 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG "result": [ 20:39:29 DEBUG { 20:39:29 DEBUG "metric": { 20:39:29 DEBUG "pod": "ds-cts-0" 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG 20:39:29 DEBUG { 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG [...] 20:39:29 DEBUG "data": { 20:39:29 DEBUG 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG "result": [ 20:39:29 DEBUG 20:39:29 DEBUG { 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG { 20:39:29 DEBUG "pod": "ds-cts-2" 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG "metric": { 20:39:29 DEBUG }, 20:39:29 DEBUG "data": { 20:39:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:29 DEBUG "value": [ 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG }, 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG "result": [ 20:39:29 DEBUG "value": [ 20:39:29 DEBUG { 20:39:29 DEBUG "0" 20:39:29 DEBUG { 20:39:29 DEBUG 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG ] 20:39:29 DEBUG "metric": { 20:39:29 DEBUG 20:39:29 DEBUG "data": { 20:39:29 DEBUG } 20:39:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:39:29 DEBUG 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG ] 20:39:29 DEBUG }, 20:39:29 DEBUG [...] 20:39:29 DEBUG "result": [ 20:39:29 DEBUG } 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG { 20:39:29 DEBUG } 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "metric": { 20:39:29 DEBUG 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 DEBUG 20:39:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:39:29 DEBUG 20:39:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:39:29 DEBUG }, 20:39:29 DEBUG [...] 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "87.1" 20:39:29 DEBUG 20:39:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:39:29 DEBUG ] 20:39:29 DEBUG [...] 20:39:29 DEBUG 20:39:29 DEBUG }, 20:39:29 DEBUG } 20:39:29 DEBUG 20:39:29 DEBUG "value": [ 20:39:29 DEBUG ] 20:39:29 DEBUG 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG } 20:39:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:39:29 DEBUG "85.76666666666667" 20:39:29 DEBUG } 20:39:29 DEBUG }, 20:39:29 DEBUG ] 20:39:29 DEBUG "value": [ 20:39:29 DEBUG } 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG ] 20:39:29 DEBUG "0.022" 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO 20:39:29 INFO [loop_until]: kubectl --namespace=xlou top node 20:39:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG { 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555569" 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713555569" 20:39:29 DEBUG "status": "success", 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555569" 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555569" 20:39:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555569" 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG "data": { 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG --- status code --- 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG "result": [ 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG { 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG "metric": { 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG "pod": "ds-idrepo-0" 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:29 DEBUG }, 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG { 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG { 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG "value": [ 20:39:29 DEBUG { 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG --- http response --- 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG { 20:39:29 DEBUG 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG "data": { 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG "data": { 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG 20:39:29 DEBUG "data": { 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG "data": { 20:39:29 DEBUG 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG "result": [ 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG { 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG "result": [ 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG [...] 20:39:29 DEBUG "result": [ 20:39:29 DEBUG { 20:39:29 DEBUG "status": "success", 20:39:29 INFO [http_cmd]: http status code OK 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:39:29 DEBUG { 20:39:29 DEBUG "result": [ 20:39:29 DEBUG 20:39:29 DEBUG { 20:39:29 DEBUG "metric": { 20:39:29 DEBUG { 20:39:29 DEBUG "metric": { 20:39:29 DEBUG "data": { 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG "metric": { 20:39:29 DEBUG { 20:39:29 DEBUG 20:39:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:39:29 DEBUG "data": { 20:39:29 DEBUG "pod": "ds-cts-0" 20:39:29 DEBUG --- status code --- 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG { 20:39:29 DEBUG { 20:39:29 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 20:39:29 DEBUG "metric": { 20:39:29 DEBUG "instance": "10.142.0.107:9100" 20:39:29 DEBUG }, 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG "result": [ 20:39:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:39:29 DEBUG "result": [ 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG }, 20:39:29 DEBUG 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG }, 20:39:29 DEBUG { 20:39:29 DEBUG --- http response --- 20:39:29 DEBUG { 20:39:29 DEBUG "data": { 20:39:29 DEBUG "data": { 20:39:29 DEBUG "value": [ 20:39:29 DEBUG "pod": "ds-cts-2" 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG "value": [ 20:39:29 DEBUG "metric": { 20:39:29 DEBUG "metric": { 20:39:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG 20:39:29 DEBUG }, 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:39:29 DEBUG "instance": "10.142.0.107:9100" 20:39:29 DEBUG ----- output ----- 20:39:29 DEBUG "result": [ 20:39:29 DEBUG "result": [ 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG }, 20:39:29 DEBUG }, 20:39:29 DEBUG { 20:39:29 DEBUG { 20:39:29 DEBUG { 20:39:29 DEBUG "value": [ 20:39:29 DEBUG [...] 20:39:29 DEBUG 20:39:29 DEBUG [...] 20:39:29 DEBUG 20:39:29 DEBUG "value": [ 20:39:29 DEBUG "value": [ 20:39:29 DEBUG "status": "success", 20:39:29 DEBUG "metric": { 20:39:29 DEBUG "metric": { 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG 20:39:29 DEBUG [...] 20:39:29 DEBUG 20:39:29 DEBUG [...] 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "data": { 20:39:29 DEBUG "pod": "ds-idrepo-0" 20:39:29 DEBUG "pod": "ds-cts-0" 20:39:29 DEBUG "0" 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "resultType": "vector", 20:39:29 DEBUG }, 20:39:29 DEBUG }, 20:39:29 DEBUG ] 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG [...] 20:39:29 DEBUG "result": [ 20:39:29 DEBUG "value": [ 20:39:29 DEBUG } 20:39:29 DEBUG "pod": "idm-b85f9d978-txpwv" 20:39:29 DEBUG 20:39:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:39:29 DEBUG 20:39:29 DEBUG [...] 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG { 20:39:29 DEBUG 20:39:29 DEBUG ] 20:39:29 DEBUG }, 20:39:29 DEBUG "instance": "10.142.0.43:9100" 20:39:29 DEBUG }, 20:39:29 DEBUG "pod": "ds-cts-2" 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "metric": { 20:39:29 DEBUG 20:39:29 DEBUG } 20:39:29 DEBUG "value": [ 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG }, 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "instance": "10.142.0.107:9100" 20:39:29 DEBUG 20:39:29 DEBUG } 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "instance": "10.142.0.43:9100" 20:39:29 DEBUG }, 20:39:29 DEBUG }, 20:39:29 DEBUG "value": [ 20:39:29 DEBUG "0" 20:39:29 DEBUG "228247.5" 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:39:29 DEBUG [...] 20:39:29 DEBUG [...] 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG 20:39:29 DEBUG ] 20:39:29 DEBUG ] 20:39:29 DEBUG "0" 20:39:29 DEBUG }, 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG "2076503237632" 20:39:29 DEBUG 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG "value": [ 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG "24354127.03333333" 20:39:29 DEBUG ] 20:39:29 DEBUG 20:39:29 DEBUG ] 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG 20:39:29 DEBUG 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG [...] 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG "8.03843118200909" 20:39:29 DEBUG "pod": "ds-idrepo-2" 20:39:29 DEBUG "pod": "ds-cts-2" 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG }, 20:39:29 DEBUG }, 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 DEBUG "value": [ 20:39:29 DEBUG "value": [ 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 DEBUG 20:39:29 DEBUG ] 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG } 20:39:29 DEBUG "instance": "10.142.0.43:9100" 20:39:29 DEBUG } 20:39:29 DEBUG "0" 20:39:29 DEBUG "0" 20:39:29 DEBUG }, 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG ] 20:39:29 DEBUG "value": [ 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 DEBUG 1713555569, 20:39:29 DEBUG ] 20:39:29 DEBUG ] 20:39:29 DEBUG "24353025.566666666" 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 DEBUG ] 20:39:29 DEBUG } 20:39:29 DEBUG } 20:39:29 INFO [loop_until]: OK (rc = 0) 20:39:29 DEBUG --- stdout --- 20:39:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 6185Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 597m 1% 7320Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2146m 7% 11080Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3376Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 5347m 17% 13974Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5173Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2744m 9% 10468Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 868m 2% 3403Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 3005m 10% 10288Mi 9% gke-xlou-cdm-ds-0f601751-j488 810m 2% 6744Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 4679m 15% 16079Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 818m 2% 7413Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 5531m 18% 13444Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 589m 3% 3951Mi 6% 20:39:29 DEBUG --- stderr --- 20:39:29 DEBUG 20:40:28 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555628" 20:40:28 INFO [http_cmd]: http status code OK 20:40:28 DEBUG --- status code --- 20:40:28 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:28 DEBUG --- http response --- 20:40:28 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:40:28 DEBUG ----- output ----- 20:40:28 DEBUG { 20:40:28 DEBUG "status": "success", 20:40:28 DEBUG "data": { 20:40:28 DEBUG "resultType": "vector", 20:40:28 DEBUG "result": [ 20:40:28 DEBUG { 20:40:28 DEBUG "metric": { 20:40:28 DEBUG "pod": "ds-idrepo-1" 20:40:28 DEBUG }, 20:40:28 DEBUG "value": [ 20:40:28 DEBUG 20:40:28 DEBUG 20:40:28 DEBUG 20:40:28 DEBUG [...] 20:40:28 DEBUG 20:40:28 DEBUG 20:40:28 DEBUG 20:40:28 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:40:28 DEBUG }, 20:40:28 DEBUG "value": [ 20:40:28 DEBUG 1713555628, 20:40:28 DEBUG "0" 20:40:28 DEBUG ] 20:40:28 DEBUG } 20:40:28 DEBUG ] 20:40:28 DEBUG } 20:40:28 DEBUG } 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713555629" 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555629" 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 ERROR [http_cmd]: ERROR 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG --- status code --- 20:40:29 ERROR --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 ERROR http status code is 400 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG --- http response --- 20:40:29 ERROR --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:29 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 20:40:29 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 20:40:29 DEBUG ----- output ----- 20:40:29 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "instance": "10.142.0.107:9100" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "instance": "10.142.0.43:9100" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0.01709999999998691" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555629" 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "instance": "10.142.0.107:9100" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "instance": "10.142.0.43:9100" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0.01266666666666841" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "ds-idrepo-0" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "ds-idrepo-2" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "ds-idrepo-1" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713555629" 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:40:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:40:29 DEBUG }, 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:40:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:40:29 DEBUG }, 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "112476160" 20:40:29 DEBUG "0" 20:40:29 DEBUG ] 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO 20:40:29 INFO [loop_until]: kubectl --namespace=xlou top pods 20:40:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:40:29 INFO [loop_until]: OK (rc = 0) 20:40:29 DEBUG --- stdout --- 20:40:29 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-767f6b67b5-9zw22 1315m 8308Mi am-767f6b67b5-qcqz2 1364m 7540Mi am-767f6b67b5-vbltz 1474m 7995Mi ds-cts-0 3878m 10975Mi ds-cts-1 3434m 11226Mi ds-cts-2 4067m 11028Mi ds-idrepo-0 1151m 3710Mi ds-idrepo-1 912m 3563Mi ds-idrepo-2 1058m 3525Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 6m 1175Mi idm-b85f9d978-wnhxb 3m 1217Mi lodemon-68f967fd98-vkx4v 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 737m 152Mi 20:40:29 DEBUG --- stderr --- 20:40:29 DEBUG 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "ds-idrepo-0" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "ds-idrepo-2" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "1352.4666666666667" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "ds-idrepo-1" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0.00035423388324102033" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0.03333333333333333" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0.03333333333333333" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0.03333333333333333" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555629" 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0.022" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "instance": "10.142.0.107:9100" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "instance": "10.142.0.43:9100" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "2077562469376" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "ds-cts-0" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "ds-cts-2" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0.000033333333333333335" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713555629" 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG { 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "ds-idrepo-0" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "ds-cts-2" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0.02854211152993825" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "ds-idrepo-0" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "ds-idrepo-2" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "am-767f6b67b5-9zw22" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "am-767f6b67b5-vbltz" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "140007.5" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO 20:40:29 INFO [loop_until]: kubectl --namespace=xlou top node 20:40:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555629" 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "ds-idrepo-1" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "end-user-ui-58878d7668-z8s6j" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "instance": "10.142.0.107:9100" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "instance": "10.142.0.43:9100" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "8861117.9" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "ds-cts-0" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "ds-cts-2" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713555629" 20:40:29 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713555629" 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "instance": "10.142.0.107:9100" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "instance": "10.142.0.43:9100" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "8862331.433333334" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [http_cmd]: http status code OK 20:40:29 DEBUG --- status code --- 20:40:29 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 20:40:29 DEBUG --- http response --- 20:40:29 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 20:40:29 DEBUG ----- output ----- 20:40:29 DEBUG { 20:40:29 DEBUG "status": "success", 20:40:29 DEBUG "data": { 20:40:29 DEBUG "resultType": "vector", 20:40:29 DEBUG "result": [ 20:40:29 DEBUG { 20:40:29 DEBUG "metric": { 20:40:29 DEBUG "pod": "ds-cts-0" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG [...] 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG 20:40:29 DEBUG "pod": "ds-cts-2" 20:40:29 DEBUG }, 20:40:29 DEBUG "value": [ 20:40:29 DEBUG 1713555629, 20:40:29 DEBUG "0" 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG ] 20:40:29 DEBUG } 20:40:29 DEBUG } 20:40:29 INFO [loop_until]: OK (rc = 0) 20:40:29 DEBUG --- stdout --- 20:40:29 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 6180Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1064m 3% 7271Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1297m 4% 11051Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3398Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 3886m 13% 14008Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 143m 0% 5177Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1418m 4% 10443Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 86m 0% 3363Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 1600m 5% 10255Mi 9% gke-xlou-cdm-ds-0f601751-j488 1026m 3% 6751Mi 6% gke-xlou-cdm-ds-0f601751-mc5d 3541m 11% 16055Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 1005m 3% 7373Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 4113m 13% 13470Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 1018m 6% 3997Mi 6% 20:40:29 DEBUG --- stderr --- 20:40:29 DEBUG 20:41:02 INFO Finished: True 20:41:02 INFO Waiting for threads to register finish flag 20:41:29 INFO Done. Have a nice day! :) 127.0.0.1 - - [19/Apr/2024 20:41:29] "GET /monitoring/stop HTTP/1.1" 200 - 20:41:32 WARNING File /tmp/lodemon_data-20240419_202919/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6521', ...) 127.0.0.1 - - [19/Apr/2024 20:41:37] "GET /monitoring/process HTTP/1.1" 200 -