==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-5d46798758-5h2st Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Thu, 18 Apr 2024 23:41:22 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=5d46798758 skaffold.dev/run-id=b4a7ffe8-4fad-41b3-86d4-4d2b2e37cf2e Annotations: Status: Running IP: 10.106.46.165 IPs: IP: 10.106.46.165 Controlled By: ReplicaSet/lodemon-5d46798758 Containers: lodemon: Container ID: containerd://d33a11a10a728a9c6cbeff1b6aa1f40c09dde9dc424e6d77ef303a1c26a87f97 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:1a2eb1b82876135a9f9dd8f7140893253f927bd2dd85e927623ff2af101538dd Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Thu, 18 Apr 2024 23:41:38 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-sq2xg (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-sq2xg: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 41m default-scheduler Successfully assigned xlou/lodemon-5d46798758-5h2st to gke-xlou-cdm-ds-0f601751-mc5d Normal Pulling 41m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 41m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 15.670629184s (15.670635003s including waiting) Normal Created 41m kubelet Created container lodemon Normal Started 41m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.46.165:8080 Press CTRL+C to quit 00:42:39 INFO 00:42:39 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:39 INFO [loop_until]: OK (rc = 0) 00:42:39 DEBUG --- stdout --- 00:42:39 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:39 DEBUG --- stderr --- 00:42:39 DEBUG 00:42:39 INFO 00:42:39 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:39 INFO [loop_until]: OK (rc = 0) 00:42:39 DEBUG --- stdout --- 00:42:39 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:39 DEBUG --- stderr --- 00:42:39 DEBUG 00:42:39 INFO 00:42:39 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:39 INFO [loop_until]: OK (rc = 0) 00:42:39 DEBUG --- stdout --- 00:42:39 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:39 DEBUG --- stderr --- 00:42:39 DEBUG 00:42:39 INFO 00:42:39 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:39 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:39 INFO [loop_until]: OK (rc = 0) 00:42:39 DEBUG --- stdout --- 00:42:39 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:39 DEBUG --- stderr --- 00:42:39 DEBUG 00:42:40 INFO 00:42:40 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:40 INFO [loop_until]: OK (rc = 0) 00:42:40 DEBUG --- stdout --- 00:42:40 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:40 DEBUG --- stderr --- 00:42:40 DEBUG 00:42:40 INFO 00:42:40 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:40 INFO [loop_until]: OK (rc = 0) 00:42:40 DEBUG --- stdout --- 00:42:40 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:40 DEBUG --- stderr --- 00:42:40 DEBUG 00:42:40 INFO 00:42:40 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:40 INFO [loop_until]: OK (rc = 0) 00:42:40 DEBUG --- stdout --- 00:42:40 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:40 DEBUG --- stderr --- 00:42:40 DEBUG 00:42:40 INFO 00:42:40 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:40 INFO [loop_until]: OK (rc = 0) 00:42:40 DEBUG --- stdout --- 00:42:40 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:40 DEBUG --- stderr --- 00:42:40 DEBUG 00:42:40 INFO 00:42:40 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:40 INFO [loop_until]: OK (rc = 0) 00:42:40 DEBUG --- stdout --- 00:42:40 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:40 DEBUG --- stderr --- 00:42:40 DEBUG 00:42:40 INFO 00:42:40 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:40 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:40 INFO [loop_until]: OK (rc = 0) 00:42:40 DEBUG --- stdout --- 00:42:40 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:40 DEBUG --- stderr --- 00:42:40 DEBUG 00:42:41 INFO 00:42:41 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:41 INFO [loop_until]: OK (rc = 0) 00:42:41 DEBUG --- stdout --- 00:42:41 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:41 DEBUG --- stderr --- 00:42:41 DEBUG 00:42:41 INFO 00:42:41 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:41 INFO [loop_until]: OK (rc = 0) 00:42:41 DEBUG --- stdout --- 00:42:41 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:41 DEBUG --- stderr --- 00:42:41 DEBUG 00:42:41 INFO 00:42:41 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:41 INFO [loop_until]: OK (rc = 0) 00:42:41 DEBUG --- stdout --- 00:42:41 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:41 DEBUG --- stderr --- 00:42:41 DEBUG 00:42:41 INFO 00:42:41 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:41 INFO [loop_until]: OK (rc = 0) 00:42:41 DEBUG --- stdout --- 00:42:41 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:41 DEBUG --- stderr --- 00:42:41 DEBUG 00:42:41 INFO 00:42:41 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:41 INFO [loop_until]: OK (rc = 0) 00:42:41 DEBUG --- stdout --- 00:42:41 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:41 DEBUG --- stderr --- 00:42:41 DEBUG 00:42:41 INFO 00:42:41 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:41 INFO [loop_until]: OK (rc = 0) 00:42:41 DEBUG --- stdout --- 00:42:41 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:41 DEBUG --- stderr --- 00:42:41 DEBUG 00:42:41 INFO 00:42:41 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:41 INFO [loop_until]: OK (rc = 0) 00:42:41 DEBUG --- stdout --- 00:42:41 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:41 DEBUG --- stderr --- 00:42:41 DEBUG 00:42:41 INFO 00:42:41 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:41 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:42 INFO [loop_until]: OK (rc = 0) 00:42:42 DEBUG --- stdout --- 00:42:42 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:42 DEBUG --- stderr --- 00:42:42 DEBUG 00:42:42 INFO 00:42:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:42 INFO [loop_until]: OK (rc = 0) 00:42:42 DEBUG --- stdout --- 00:42:42 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:42 DEBUG --- stderr --- 00:42:42 DEBUG 00:42:42 INFO 00:42:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:42 INFO [loop_until]: OK (rc = 0) 00:42:42 DEBUG --- stdout --- 00:42:42 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:42 DEBUG --- stderr --- 00:42:42 DEBUG 00:42:42 INFO 00:42:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:42 INFO [loop_until]: OK (rc = 0) 00:42:42 DEBUG --- stdout --- 00:42:42 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:42 DEBUG --- stderr --- 00:42:42 DEBUG 00:42:42 INFO 00:42:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:42 INFO [loop_until]: OK (rc = 0) 00:42:42 DEBUG --- stdout --- 00:42:42 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:42 DEBUG --- stderr --- 00:42:42 DEBUG 00:42:42 INFO 00:42:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:42 INFO [loop_until]: OK (rc = 0) 00:42:42 DEBUG --- stdout --- 00:42:42 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:42 DEBUG --- stderr --- 00:42:42 DEBUG 00:42:42 INFO 00:42:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:42 INFO [loop_until]: OK (rc = 0) 00:42:42 DEBUG --- stdout --- 00:42:42 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:42 DEBUG --- stderr --- 00:42:42 DEBUG 00:42:42 INFO 00:42:42 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:42 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:43 INFO [loop_until]: OK (rc = 0) 00:42:43 DEBUG --- stdout --- 00:42:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:43 DEBUG --- stderr --- 00:42:43 DEBUG 00:42:43 INFO 00:42:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:43 INFO [loop_until]: OK (rc = 0) 00:42:43 DEBUG --- stdout --- 00:42:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:43 DEBUG --- stderr --- 00:42:43 DEBUG 00:42:43 INFO 00:42:43 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 00:42:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:43 INFO [loop_until]: OK (rc = 0) 00:42:43 DEBUG --- stdout --- 00:42:43 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 00:42:43 DEBUG --- stderr --- 00:42:43 DEBUG 00:42:43 INFO Initializing monitoring instance threads 00:42:43 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 00:42:43 INFO Starting instance threads 00:42:43 INFO 00:42:43 INFO Thread started 00:42:43 INFO [loop_until]: kubectl --namespace=xlou top node 00:42:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:43 INFO 00:42:43 INFO [loop_until]: kubectl --namespace=xlou top pods 00:42:43 INFO Thread started 00:42:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 DEBUG --- status code --- 00:42:43 INFO Thread started 00:42:43 DEBUG --- status code --- 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 INFO Thread started 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG --- http response --- 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 INFO Thread started 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG --- http response --- 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 DEBUG ----- output ----- 00:42:43 INFO Thread started 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG { 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:43 DEBUG { 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG { 00:42:43 DEBUG { 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG "data": { 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:43 DEBUG { 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG { 00:42:43 DEBUG { 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG "resultType": "vector", 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 DEBUG { 00:42:43 INFO Thread started 00:42:43 DEBUG { 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG { 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:43 DEBUG "data": { 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG "data": { 00:42:43 DEBUG "data": { 00:42:43 DEBUG { 00:42:43 DEBUG { 00:42:43 DEBUG "result": [ 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG "status": "success", 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 DEBUG { 00:42:43 DEBUG "data": { 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "data": { 00:42:43 DEBUG "data": { 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG { 00:42:43 DEBUG "data": { 00:42:43 DEBUG "data": { 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 DEBUG "status": "success", 00:42:43 INFO Thread started 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "data": { 00:42:43 DEBUG { 00:42:43 INFO [loop_until]: OK (rc = 0) 00:42:43 DEBUG "result": [ 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "result": [ 00:42:43 DEBUG "result": [ 00:42:43 DEBUG "data": { 00:42:43 DEBUG "data": { 00:42:43 DEBUG "metric": { 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "data": { 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 DEBUG "result": [ 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 DEBUG "resultType": "vector", 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG --- stdout --- 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [loop_until]: OK (rc = 0) 00:42:43 DEBUG { 00:42:43 DEBUG "result": [ 00:42:43 DEBUG "result": [ 00:42:43 DEBUG { 00:42:43 DEBUG { 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:42:43 DEBUG "result": [ 00:42:43 DEBUG "result": [ 00:42:43 DEBUG "resultType": "vector", 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG { 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG "result": [ 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG "data": { 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 981m 4181Mi am-6fd554f8f7-hvvkw 1034m 4083Mi am-6fd554f8f7-pmj4b 404m 3486Mi ds-cts-0 2086m 9674Mi ds-cts-1 1973m 9731Mi ds-cts-2 1840m 9803Mi ds-idrepo-0 165m 3264Mi ds-idrepo-1 184m 2945Mi ds-idrepo-2 175m 3022Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 55Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1m 269Mi 00:42:43 DEBUG --- stdout --- 00:42:43 DEBUG "metric": { 00:42:43 DEBUG { 00:42:43 DEBUG { 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 DEBUG "metric": { 00:42:43 DEBUG "metric": { 00:42:43 DEBUG "result": [ 00:42:43 DEBUG "result": [ 00:42:43 DEBUG }, 00:42:43 DEBUG { 00:42:43 DEBUG { 00:42:43 DEBUG "result": [ 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483763" 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 INFO Thread started 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG "metric": { 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG { 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG --- stderr --- 00:42:43 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 6048Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 267m 0% 6656Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1019m 3% 6810Mi 6% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 3216Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1925m 6% 12047Mi 10% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 5205Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1046m 3% 6922Mi 6% gke-xlou-cdm-ds-0f601751-cf7b 61m 0% 3103Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 437m 1% 5653Mi 5% gke-xlou-cdm-ds-0f601751-j488 235m 0% 6073Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 1886m 6% 13585Mi 12% gke-xlou-cdm-ds-0f601751-nj6q 277m 0% 6560Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 1952m 6% 11611Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 69m 0% 4053Mi 6% 00:42:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:42:43 DEBUG "metric": { 00:42:43 DEBUG "metric": { 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:42:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:42:43 DEBUG { 00:42:43 DEBUG { 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "metric": { 00:42:43 DEBUG "metric": { 00:42:43 DEBUG { 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG --- http response --- 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483763" 00:42:43 DEBUG "instance": "10.142.0.107:9100" 00:42:43 INFO Thread started 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG "metric": { 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG "result": [ 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG 00:42:43 DEBUG --- stderr --- 00:42:43 DEBUG }, 00:42:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:42:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG }, 00:42:43 DEBUG }, 00:42:43 DEBUG "metric": { 00:42:43 DEBUG "metric": { 00:42:43 DEBUG 00:42:43 DEBUG "instance": "10.142.0.107:9100" 00:42:43 DEBUG "pod": "ds-cts-0" 00:42:43 DEBUG "metric": { 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:43 DEBUG }, 00:42:43 ERROR [http_cmd]: ERROR 00:42:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483763" 00:42:43 INFO Thread started 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 DEBUG "pod": "ds-idrepo-0" 00:42:43 DEBUG { 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:42:43 DEBUG 00:42:43 DEBUG "value": [ 00:42:43 DEBUG }, 00:42:43 DEBUG }, 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "instance": "10.142.0.107:9100" 00:42:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:42:43 DEBUG 00:42:43 DEBUG }, 00:42:43 DEBUG }, 00:42:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG "value": [ 00:42:43 ERROR --- status code --- 00:42:43 INFO All threads has been started 127.0.0.1 - - [19/Apr/2024 00:42:43] "GET /monitoring/start HTTP/1.1" 200 - 00:42:43 DEBUG ----- output ----- 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG }, 00:42:43 DEBUG "metric": { 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "value": [ 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG }, 00:42:43 DEBUG }, 00:42:43 DEBUG 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "value": [ 00:42:43 DEBUG }, 00:42:43 DEBUG { 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG { 00:42:43 DEBUG { 00:42:43 DEBUG 00:42:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:42:43 DEBUG { 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "pod": "ds-idrepo-0" 00:42:43 DEBUG { 00:42:43 DEBUG { 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "value": [ 00:42:43 DEBUG [...] 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG { 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG 00:42:43 ERROR --- http response --- 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG 00:42:43 DEBUG }, 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG { 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG "data": { 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG "data": { 00:42:43 DEBUG "data": { 00:42:43 DEBUG 00:42:43 DEBUG "data": { 00:42:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "data": { 00:42:43 DEBUG "data": { 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG [...] 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG [...] 00:42:43 DEBUG [...] 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "data": { 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG [...] 00:42:43 DEBUG "resultType": "vector", 00:42:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:42:43 DEBUG 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG 00:42:43 DEBUG [...] 00:42:43 DEBUG [...] 00:42:43 INFO [http_cmd]: http status code OK 00:42:43 DEBUG "data": { 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG [...] 00:42:43 DEBUG [...] 00:42:43 DEBUG 00:42:43 DEBUG "result": [ 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "result": [ 00:42:43 DEBUG "result": [ 00:42:43 DEBUG 00:42:43 DEBUG "result": [ 00:42:43 DEBUG [...] 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG { 00:42:43 DEBUG 00:42:43 DEBUG "result": [ 00:42:43 DEBUG "result": [ 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:42:43 DEBUG 00:42:43 DEBUG --- status code --- 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG [...] 00:42:43 DEBUG [...] 00:42:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG [...] 00:42:43 DEBUG { 00:42:43 DEBUG "result": [ 00:42:43 DEBUG { 00:42:43 DEBUG { 00:42:43 DEBUG 00:42:43 DEBUG { 00:42:43 DEBUG 00:42:43 DEBUG { 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG 00:42:43 DEBUG { 00:42:43 DEBUG { 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:42:43 DEBUG "result": [ 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG }, 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG "metric": { 00:42:43 DEBUG { 00:42:43 DEBUG "metric": { 00:42:43 DEBUG "metric": { 00:42:43 DEBUG 00:42:43 DEBUG "metric": { 00:42:43 DEBUG 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG "data": { 00:42:43 DEBUG [...] 00:42:43 DEBUG "metric": { 00:42:43 DEBUG "metric": { 00:42:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG --- http response --- 00:42:43 DEBUG { 00:42:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:42:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG "value": [ 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG "pod": "ds-cts-0" 00:42:43 DEBUG "metric": { 00:42:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:42:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:42:43 DEBUG "instance": "10.142.0.43:9100" 00:42:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:42:43 DEBUG 00:42:43 DEBUG "data": { 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG 00:42:43 DEBUG "pod": "ds-idrepo-0" 00:42:43 DEBUG "pod": "ds-cts-0" 00:42:43 DEBUG }, 00:42:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:42:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:42:43 DEBUG "metric": { 00:42:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:42:43 DEBUG }, 00:42:43 DEBUG }, 00:42:43 DEBUG 00:42:43 DEBUG "instance": "10.142.0.43:9100" 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG "instance": "10.142.0.43:9100" 00:42:43 DEBUG "pod": "ds-cts-2" 00:42:43 DEBUG 00:42:43 DEBUG }, 00:42:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:42:43 DEBUG }, 00:42:43 DEBUG }, 00:42:43 DEBUG }, 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "pod": "ds-idrepo-2" 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG "result": [ 00:42:43 DEBUG 00:42:43 DEBUG }, 00:42:43 DEBUG }, 00:42:43 DEBUG "value": [ 00:42:43 DEBUG }, 00:42:43 DEBUG }, 00:42:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:42:43 DEBUG ----- output ----- 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "value": [ 00:42:43 DEBUG 00:42:43 DEBUG }, 00:42:43 DEBUG "0.03464574408231773" 00:42:43 DEBUG }, 00:42:43 DEBUG }, 00:42:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:42:43 DEBUG "value": [ 00:42:43 DEBUG }, 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "value": [ 00:42:43 DEBUG }, 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG }, 00:42:43 DEBUG "result": [ 00:42:43 DEBUG { 00:42:43 DEBUG 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "value": [ 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "value": [ 00:42:43 DEBUG }, 00:42:43 DEBUG { 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:42:43 DEBUG "value": [ 00:42:43 DEBUG ] 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "value": [ 00:42:43 DEBUG }, 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "5005007.566666666" 00:42:43 DEBUG "value": [ 00:42:43 DEBUG { 00:42:43 DEBUG "metric": { 00:42:43 DEBUG "pod": "ds-idrepo-2" 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG "0.352593612774451" 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "status": "success", 00:42:43 DEBUG "639744.510978044" 00:42:43 DEBUG "545.0432468396541" 00:42:43 DEBUG }, 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG } 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "value": [ 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG ] 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG "metric": { 00:42:43 DEBUG "pod": "ds-idrepo-0" 00:42:43 DEBUG }, 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG ] 00:42:43 DEBUG "380411904" 00:42:43 DEBUG "0" 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG ] 00:42:43 DEBUG ] 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "0.0029999999999999953" 00:42:43 DEBUG ] 00:42:43 DEBUG "4998610.666666667" 00:42:43 DEBUG "0" 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG [...] 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG "0" 00:42:43 DEBUG "instance": "10.142.0.107:9100" 00:42:43 DEBUG }, 00:42:43 DEBUG "value": [ 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG ] 00:42:43 DEBUG ] 00:42:43 DEBUG "data": { 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG ] 00:42:43 DEBUG ] 00:42:43 DEBUG "0" 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG ] 00:42:43 DEBUG }, 00:42:43 DEBUG "value": [ 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG [...] 00:42:43 DEBUG 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG "resultType": "vector", 00:42:43 DEBUG [...] 00:42:43 DEBUG ] 00:42:43 DEBUG ] 00:42:43 DEBUG "0" 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG ] 00:42:43 DEBUG [...] 00:42:43 DEBUG 00:42:43 DEBUG [...] 00:42:43 DEBUG ] 00:42:43 DEBUG "pod": "ds-cts-2" 00:42:43 DEBUG [...] 00:42:43 DEBUG } 00:42:43 DEBUG "value": [ 00:42:43 DEBUG 00:42:43 DEBUG "0" 00:42:43 DEBUG [...] 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG ] 00:42:43 DEBUG ] 00:42:43 DEBUG "result": [ 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG ] 00:42:43 DEBUG ] 00:42:43 DEBUG ] 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG [...] 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG }, 00:42:43 DEBUG 00:42:43 DEBUG ] 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG ] 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG { 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG ] 00:42:43 DEBUG 00:42:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG "value": [ 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG 00:42:43 DEBUG "pod": "ds-idrepo-2" 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG "metric": { 00:42:43 DEBUG 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG 00:42:43 DEBUG }, 00:42:43 DEBUG 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG 00:42:43 DEBUG [...] 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG }, 00:42:43 DEBUG "instance": "10.142.0.107:9100" 00:42:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG 00:42:43 DEBUG "value": [ 00:42:43 DEBUG 00:42:43 DEBUG "0.03333333333333333" 00:42:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:42:43 DEBUG [...] 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG "value": [ 00:42:43 DEBUG }, 00:42:43 DEBUG }, 00:42:43 DEBUG } 00:42:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:42:43 DEBUG ] 00:42:43 DEBUG }, 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG "pod": "ds-cts-2" 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "value": [ 00:42:43 DEBUG }, 00:42:43 DEBUG "0" 00:42:43 DEBUG }, 00:42:43 DEBUG } 00:42:43 DEBUG "value": [ 00:42:43 DEBUG 00:42:43 DEBUG 00:42:43 DEBUG }, 00:42:43 DEBUG "0" 00:42:43 DEBUG 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG "value": [ 00:42:43 DEBUG ] 00:42:43 DEBUG "value": [ 00:42:43 DEBUG ] 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG 00:42:43 DEBUG "pod": "ds-cts-2" 00:42:43 DEBUG "value": [ 00:42:43 DEBUG ] 00:42:43 DEBUG 00:42:43 DEBUG "2.598" 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG } 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG } 00:42:43 DEBUG "1000" 00:42:43 DEBUG "instance": "10.142.0.43:9100" 00:42:43 DEBUG }, 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG } 00:42:43 DEBUG 00:42:43 DEBUG ] 00:42:43 DEBUG "1" 00:42:43 DEBUG ] 00:42:43 DEBUG "0" 00:42:43 DEBUG } 00:42:43 DEBUG ] 00:42:43 DEBUG }, 00:42:43 DEBUG "value": [ 00:42:43 DEBUG "0" 00:42:43 DEBUG ] 00:42:43 DEBUG [...] 00:42:43 DEBUG } 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG "value": [ 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG ] 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG "0" 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG ] 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG "272657763328" 00:42:43 DEBUG ] 00:42:43 DEBUG ] 00:42:43 DEBUG 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG "instance": "10.142.0.43:9100" 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG }, 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG "value": [ 00:42:43 DEBUG } 00:42:43 DEBUG } 00:42:43 DEBUG 1713483763, 00:42:43 DEBUG } 00:42:43 DEBUG "0.00526666666666434" 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG ] 00:42:43 DEBUG } 00:42:43 DEBUG } 00:43:43 INFO 00:43:43 INFO [loop_until]: kubectl --namespace=xlou top pods 00:43:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:43:43 INFO 00:43:43 INFO [loop_until]: kubectl --namespace=xlou top node 00:43:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 ERROR [http_cmd]: ERROR 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483823" 00:43:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713483823" 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 DEBUG --- status code --- 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [loop_until]: OK (rc = 0) 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 ERROR --- status code --- 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [loop_until]: OK (rc = 0) 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- stdout --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- stdout --- 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- http response --- 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 4304m 4464Mi am-6fd554f8f7-hvvkw 2784m 4203Mi am-6fd554f8f7-pmj4b 2386m 3690Mi ds-cts-0 2414m 9675Mi ds-cts-1 2685m 9708Mi ds-cts-2 1799m 9806Mi ds-idrepo-0 553m 3264Mi ds-idrepo-1 791m 2946Mi ds-idrepo-2 954m 3022Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 2721m 688Mi 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 ERROR --- http response --- 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5934Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1202m 4% 6664Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2847m 9% 6877Mi 6% gke-xlou-cdm-default-pool-1b0769c8-qkns 79m 0% 3203Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2986m 10% 12041Mi 10% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 5214Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2484m 8% 7139Mi 6% gke-xlou-cdm-ds-0f601751-cf7b 1945m 6% 3103Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 3740m 12% 6029Mi 5% gke-xlou-cdm-ds-0f601751-j488 741m 2% 6070Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 2841m 9% 13591Mi 12% gke-xlou-cdm-ds-0f601751-nj6q 1325m 4% 6575Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 3041m 10% 11612Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 3306m 20% 4463Mi 7% 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- status code --- 00:43:43 INFO [http_cmd]: http status code OK 00:43:43 DEBUG --- status code --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- stderr --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- stderr --- 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:43 DEBUG 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:43:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:43 DEBUG 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG --- http response --- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:43:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG { 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG ----- output ----- 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG { 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "data": { 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "data": { 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "data": { 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "data": { 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "data": { 00:43:43 DEBUG "status": "success", 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG { 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "result": [ 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG "data": { 00:43:43 DEBUG "result": [ 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "result": [ 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "result": [ 00:43:43 DEBUG { 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "resultType": "vector", 00:43:43 DEBUG { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG { 00:43:43 DEBUG }, 00:43:43 DEBUG { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "pod": "ds-cts-0" 00:43:43 DEBUG { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:43:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:43:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:43:43 DEBUG "instance": "10.142.0.107:9100" 00:43:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:43:43 DEBUG "instance": "10.142.0.107:9100" 00:43:43 DEBUG { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:43:43 DEBUG "result": [ 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:43:43 DEBUG "pod": "ds-idrepo-0" 00:43:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "pod": "ds-idrepo-0" 00:43:43 DEBUG "pod": "ds-cts-0" 00:43:43 DEBUG }, 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "pod": "ds-cts-0" 00:43:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:43:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:43:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:43:43 DEBUG }, 00:43:43 DEBUG }, 00:43:43 DEBUG }, 00:43:43 DEBUG }, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG }, 00:43:43 DEBUG "metric": { 00:43:43 DEBUG }, 00:43:43 DEBUG { 00:43:43 DEBUG { 00:43:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:43:43 DEBUG }, 00:43:43 DEBUG }, 00:43:43 DEBUG }, 00:43:43 DEBUG "instance": "10.142.0.107:9100" 00:43:43 DEBUG 00:43:43 DEBUG "pod": "ds-idrepo-0" 00:43:43 DEBUG }, 00:43:43 DEBUG }, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:43:43 DEBUG }, 00:43:43 DEBUG }, 00:43:43 DEBUG }, 00:43:43 DEBUG }, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "value": [ 00:43:43 DEBUG }, 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG "instance": "10.142.0.107:9100" 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "metric": { 00:43:43 DEBUG "metric": { 00:43:43 DEBUG }, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "value": [ 00:43:43 DEBUG }, 00:43:43 DEBUG 00:43:43 DEBUG }, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 00:43:43 DEBUG }, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 00:43:43 DEBUG }, 00:43:43 DEBUG 00:43:43 DEBUG "instance": "10.142.0.107:9100" 00:43:43 DEBUG "pod": "ds-idrepo-0" 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG }, 00:43:43 DEBUG }, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG "value": [ 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG "instance": "10.142.0.43:9100" 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:43:43 DEBUG 00:43:43 DEBUG }, 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG [...] 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:43:43 DEBUG }, 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG "pod": "ds-cts-2" 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG 00:43:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:43:43 DEBUG 00:43:43 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:43:43 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:43:43 DEBUG 00:43:43 DEBUG }, 00:43:43 DEBUG 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 00:43:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:43:43 DEBUG "instance": "10.142.0.43:9100" 00:43:43 DEBUG 00:43:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:43:43 DEBUG 00:43:43 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:43:43 DEBUG 00:43:43 DEBUG }, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 00:43:43 DEBUG "pod": "ds-idrepo-2" 00:43:43 DEBUG "pod": "ds-cts-2" 00:43:43 DEBUG }, 00:43:43 DEBUG 00:43:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:43:43 DEBUG 00:43:43 DEBUG }, 00:43:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:43:43 DEBUG }, 00:43:43 DEBUG }, 00:43:43 DEBUG 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG "32695857.03333333" 00:43:43 DEBUG ] 00:43:43 DEBUG }, 00:43:43 DEBUG 00:43:43 DEBUG }, 00:43:43 DEBUG "pod": "ds-idrepo-2" 00:43:43 DEBUG }, 00:43:43 DEBUG 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG "pod": "ds-idrepo-2" 00:43:43 DEBUG }, 00:43:43 DEBUG }, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "pod": "ds-cts-2" 00:43:43 DEBUG }, 00:43:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:43:43 DEBUG "value": [ 00:43:43 DEBUG }, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "instance": "10.142.0.43:9100" 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG }, 00:43:43 DEBUG [...] 00:43:43 DEBUG }, 00:43:43 DEBUG } 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 00:43:43 DEBUG "value": [ 00:43:43 DEBUG }, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "instance": "10.142.0.43:9100" 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG "0" 00:43:43 DEBUG }, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG }, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG }, 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG }, 00:43:43 DEBUG "0.02596429152443666" 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 00:43:43 DEBUG "value": [ 00:43:43 DEBUG ] 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG "pod": "ds-cts-2" 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG }, 00:43:43 DEBUG "636.1822916666666" 00:43:43 DEBUG ] 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG "0" 00:43:43 DEBUG "value": [ 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG "value": [ 00:43:43 DEBUG "1063.9" 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG "4.0055" 00:43:43 DEBUG "35941382.907699294" 00:43:43 DEBUG "value": [ 00:43:43 DEBUG ] 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG } 00:43:43 DEBUG "637.0681616666668" 00:43:43 DEBUG }, 00:43:43 DEBUG "0" 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG "345.202973341117" 00:43:43 DEBUG "value": [ 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG "0" 00:43:43 DEBUG "0" 00:43:43 DEBUG ] 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG "16038" 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG ] 00:43:43 DEBUG "0.002047282868262531" 00:43:43 DEBUG ] 00:43:43 DEBUG ] 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG } 00:43:43 DEBUG "380227584" 00:43:43 DEBUG 00:43:43 DEBUG "0.037466666666663664" 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG "value": [ 00:43:43 DEBUG ] 00:43:43 DEBUG "0" 00:43:43 DEBUG ] 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG "0" 00:43:43 DEBUG ] 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG "0.00002766" 00:43:43 DEBUG ] 00:43:43 DEBUG "833.3333333333333" 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG "0.02699999999999978" 00:43:43 DEBUG ] 00:43:43 DEBUG ] 00:43:43 DEBUG "instance": "10.142.0.43:9100" 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG "32700339.4" 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG ] 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG }, 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG "0" 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG "value": [ 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG 1713483823, 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG "273130447872" 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG } 00:43:43 DEBUG ] 00:43:43 DEBUG } 00:43:43 DEBUG } 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713483883" 00:44:43 INFO [loop_until]: kubectl --namespace=xlou top pods 00:44:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:44:43 INFO 00:44:43 INFO [loop_until]: kubectl --namespace=xlou top node 00:44:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483883" 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:44:43 ERROR [http_cmd]: ERROR 00:44:43 ERROR --- status code --- 00:44:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:44:43 ERROR --- http response --- 00:44:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:44:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 DEBUG --- status code --- 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:43 DEBUG { 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG { 00:44:43 DEBUG "data": { 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "data": { 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG { 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "metric": { 00:44:43 DEBUG { 00:44:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:44:43 DEBUG "metric": { 00:44:43 DEBUG }, 00:44:43 DEBUG "instance": "10.142.0.107:9100" 00:44:43 DEBUG "value": [ 00:44:43 DEBUG }, 00:44:43 DEBUG 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG "instance": "10.142.0.43:9100" 00:44:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:44:43 DEBUG }, 00:44:43 DEBUG }, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "275151055872" 00:44:43 DEBUG "555.3145336225596" 00:44:43 DEBUG ] 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 DEBUG ] 00:44:43 DEBUG ] 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG { 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "data": { 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "result": [ 00:44:43 DEBUG { 00:44:43 DEBUG "metric": { 00:44:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:44:43 DEBUG }, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:44:43 DEBUG }, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "758.7666666666667" 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG { 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "data": { 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "result": [ 00:44:43 DEBUG { 00:44:43 DEBUG "metric": { 00:44:43 DEBUG "instance": "10.142.0.107:9100" 00:44:43 DEBUG }, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG "instance": "10.142.0.43:9100" 00:44:43 DEBUG }, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "0.044033333333334215" 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483883" 00:44:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713483883" 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 INFO [loop_until]: OK (rc = 0) 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 INFO [loop_until]: OK (rc = 0) 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG --- status code --- 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 DEBUG --- stdout --- 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 DEBUG --- status code --- 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG --- stdout --- 00:44:43 DEBUG --- status code --- 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10602m 5390Mi am-6fd554f8f7-hvvkw 9229m 5173Mi am-6fd554f8f7-pmj4b 9403m 4870Mi ds-cts-0 7245m 9798Mi ds-cts-1 8244m 9919Mi ds-cts-2 7636m 10062Mi ds-idrepo-0 4720m 3265Mi ds-idrepo-1 4721m 2948Mi ds-idrepo-2 4626m 3022Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1172Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7125m 871Mi 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 5926Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4648m 15% 6670Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9381m 31% 7877Mi 7% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3208Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 7759m 25% 12349Mi 11% gke-xlou-cdm-default-pool-1b0769c8-wpwl 165m 0% 5201Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10604m 35% 8155Mi 7% gke-xlou-cdm-ds-0f601751-cf7b 27916m 93% 3143Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10312m 34% 7131Mi 6% gke-xlou-cdm-ds-0f601751-j488 5066m 16% 6064Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8267m 27% 13849Mi 12% gke-xlou-cdm-ds-0f601751-nj6q 5293m 17% 6579Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 7477m 25% 11896Mi 10% gke-xlou-cdm-frontend-a8771548-i82n 7140m 44% 4642Mi 7% 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG --- stderr --- 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG --- http response --- 00:44:43 INFO [http_cmd]: http status code OK 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG --- stderr --- 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:43 DEBUG 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:43 DEBUG --- status code --- 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:43 DEBUG 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG --- http response --- 00:44:43 DEBUG { 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG { 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:43 DEBUG { 00:44:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG { 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG { 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG ----- output ----- 00:44:43 DEBUG "data": { 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "data": { 00:44:43 DEBUG "data": { 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG "data": { 00:44:43 DEBUG "data": { 00:44:43 DEBUG "data": { 00:44:43 DEBUG "data": { 00:44:43 DEBUG "data": { 00:44:43 DEBUG "data": { 00:44:43 DEBUG "data": { 00:44:43 DEBUG "data": { 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "data": { 00:44:43 DEBUG { 00:44:43 DEBUG "data": { 00:44:43 DEBUG { 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "data": { 00:44:43 DEBUG "data": { 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "data": { 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "data": { 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "status": "success", 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "data": { 00:44:43 DEBUG "data": { 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG { 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "data": { 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "data": { 00:44:43 DEBUG { 00:44:43 DEBUG "result": [ 00:44:43 DEBUG { 00:44:43 DEBUG "result": [ 00:44:43 DEBUG { 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "metric": { 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG "result": [ 00:44:43 DEBUG { 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG { 00:44:43 DEBUG "metric": { 00:44:43 DEBUG "metric": { 00:44:43 DEBUG { 00:44:43 DEBUG "metric": { 00:44:43 DEBUG { 00:44:43 DEBUG "metric": { 00:44:43 DEBUG { 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "metric": { 00:44:43 DEBUG "metric": { 00:44:43 DEBUG "metric": { 00:44:43 DEBUG "metric": { 00:44:43 DEBUG { 00:44:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:44:43 DEBUG "metric": { 00:44:43 DEBUG "metric": { 00:44:43 DEBUG { 00:44:43 DEBUG "metric": { 00:44:43 DEBUG "result": [ 00:44:43 DEBUG "pod": "ds-cts-0" 00:44:43 DEBUG "resultType": "vector", 00:44:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:44:43 DEBUG "metric": { 00:44:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:44:43 DEBUG "metric": { 00:44:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:44:43 DEBUG "metric": { 00:44:43 DEBUG { 00:44:43 DEBUG { 00:44:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:44:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:44:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:44:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:44:43 DEBUG "metric": { 00:44:43 DEBUG }, 00:44:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:44:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:44:43 DEBUG "metric": { 00:44:43 DEBUG "pod": "ds-idrepo-0" 00:44:43 DEBUG { 00:44:43 DEBUG }, 00:44:43 DEBUG "result": [ 00:44:43 DEBUG }, 00:44:43 DEBUG "pod": "ds-idrepo-0" 00:44:43 DEBUG }, 00:44:43 DEBUG "instance": "10.142.0.107:9100" 00:44:43 DEBUG }, 00:44:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:44:43 DEBUG "metric": { 00:44:43 DEBUG "metric": { 00:44:43 DEBUG }, 00:44:43 DEBUG }, 00:44:43 DEBUG }, 00:44:43 DEBUG }, 00:44:43 DEBUG "pod": "ds-cts-0" 00:44:43 DEBUG "value": [ 00:44:43 DEBUG }, 00:44:43 DEBUG }, 00:44:43 DEBUG "pod": "ds-cts-0" 00:44:43 DEBUG }, 00:44:43 DEBUG "metric": { 00:44:43 DEBUG "value": [ 00:44:43 DEBUG { 00:44:43 DEBUG "value": [ 00:44:43 DEBUG }, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG }, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG }, 00:44:43 DEBUG "pod": "ds-idrepo-0" 00:44:43 DEBUG "pod": "ds-idrepo-0" 00:44:43 DEBUG "value": [ 00:44:43 DEBUG "value": [ 00:44:43 DEBUG "value": [ 00:44:43 DEBUG "value": [ 00:44:43 DEBUG }, 00:44:43 DEBUG 00:44:43 DEBUG "value": [ 00:44:43 DEBUG "value": [ 00:44:43 DEBUG }, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG "instance": "10.142.0.107:9100" 00:44:43 DEBUG 00:44:43 DEBUG "metric": { 00:44:43 DEBUG 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 00:44:43 DEBUG "value": [ 00:44:43 DEBUG }, 00:44:43 DEBUG }, 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 00:44:43 DEBUG }, 00:44:43 DEBUG 00:44:43 DEBUG "instance": "10.142.0.107:9100" 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG "value": [ 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 00:44:43 DEBUG }, 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG "value": [ 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG 00:44:43 DEBUG "pod": "ds-cts-2" 00:44:43 DEBUG 00:44:43 DEBUG [...] 00:44:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:44:43 DEBUG 00:44:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:44:43 DEBUG 00:44:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG 00:44:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:44:43 DEBUG 00:44:43 DEBUG }, 00:44:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:44:43 DEBUG 00:44:43 DEBUG }, 00:44:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:44:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:44:43 DEBUG 00:44:43 DEBUG "pod": "ds-cts-2" 00:44:43 DEBUG }, 00:44:43 DEBUG "pod": "ds-idrepo-2" 00:44:43 DEBUG 00:44:43 DEBUG }, 00:44:43 DEBUG "pod": "ds-idrepo-2" 00:44:43 DEBUG }, 00:44:43 DEBUG "instance": "10.142.0.43:9100" 00:44:43 DEBUG }, 00:44:43 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:44:43 DEBUG 00:44:43 DEBUG "pod": "ds-idrepo-2" 00:44:43 DEBUG }, 00:44:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "6.1785928684279625" 00:44:43 DEBUG "value": [ 00:44:43 DEBUG }, 00:44:43 DEBUG }, 00:44:43 DEBUG 00:44:43 DEBUG }, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG }, 00:44:43 DEBUG 00:44:43 DEBUG "value": [ 00:44:43 DEBUG }, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG }, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG }, 00:44:43 DEBUG 00:44:43 DEBUG }, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "pod": "ds-cts-2" 00:44:43 DEBUG }, 00:44:43 DEBUG ] 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG "pod": "ds-cts-2" 00:44:43 DEBUG "value": [ 00:44:43 DEBUG }, 00:44:43 DEBUG "117915.5" 00:44:43 DEBUG }, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG } 00:44:43 DEBUG "16308338.394793924" 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "instance": "10.142.0.43:9100" 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "0" 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "instance": "10.142.0.43:9100" 00:44:43 DEBUG "1912.7" 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "0" 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "1912.5" 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG }, 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "value": [ 00:44:43 DEBUG ] 00:44:43 DEBUG "value": [ 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG ] 00:44:43 DEBUG ] 00:44:43 DEBUG "1913.2" 00:44:43 DEBUG "0" 00:44:43 DEBUG }, 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG "0" 00:44:43 DEBUG }, 00:44:43 DEBUG ] 00:44:43 DEBUG "0" 00:44:43 DEBUG ] 00:44:43 DEBUG "50422812.13333333" 00:44:43 DEBUG ] 00:44:43 DEBUG "1008566272" 00:44:43 DEBUG "value": [ 00:44:43 DEBUG "0" 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG } 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG "0" 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG ] 00:44:43 DEBUG "value": [ 00:44:43 DEBUG "0" 00:44:43 DEBUG ] 00:44:43 DEBUG ] 00:44:43 DEBUG "value": [ 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG ] 00:44:43 DEBUG "5.733" 00:44:43 DEBUG ] 00:44:43 DEBUG "9.486678619848153" 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG 1713483883, 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG "0" 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG ] 00:44:43 DEBUG "0.03166666666666632" 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG "50426826.13333333" 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG ] 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG ] 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:44:43 DEBUG } 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483943" 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 DEBUG --- status code --- 00:45:43 ERROR [http_cmd]: ERROR 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 ERROR --- status code --- 00:45:43 DEBUG --- http response --- 00:45:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:45:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:45:43 ERROR --- http response --- 00:45:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:45:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG { 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "data": { 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "result": [ 00:45:43 DEBUG { 00:45:43 DEBUG "metric": { 00:45:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:45:43 DEBUG }, 00:45:43 DEBUG "value": [ 00:45:43 DEBUG 00:45:43 DEBUG 00:45:43 DEBUG 00:45:43 DEBUG [...] 00:45:43 DEBUG 00:45:43 DEBUG 00:45:43 DEBUG 00:45:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:45:43 DEBUG }, 00:45:43 DEBUG "value": [ 00:45:43 DEBUG 1713483943, 00:45:43 DEBUG "666.6666666666666" 00:45:43 DEBUG ] 00:45:43 DEBUG } 00:45:43 DEBUG ] 00:45:43 DEBUG } 00:45:43 DEBUG } 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG --- http response --- 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG { 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:45:43 DEBUG "data": { 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG { 00:45:43 DEBUG "result": [ 00:45:43 DEBUG { 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG { 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "data": { 00:45:43 DEBUG "metric": { 00:45:43 DEBUG "data": { 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "instance": "10.142.0.107:9100" 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "result": [ 00:45:43 DEBUG }, 00:45:43 DEBUG "result": [ 00:45:43 DEBUG { 00:45:43 DEBUG "value": [ 00:45:43 DEBUG { 00:45:43 DEBUG "metric": { 00:45:43 DEBUG 00:45:43 DEBUG "metric": { 00:45:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:45:43 DEBUG 00:45:43 DEBUG "instance": "10.142.0.107:9100" 00:45:43 DEBUG }, 00:45:43 DEBUG 00:45:43 DEBUG }, 00:45:43 DEBUG "value": [ 00:45:43 DEBUG [...] 00:45:43 DEBUG "value": [ 00:45:43 DEBUG 00:45:43 DEBUG 00:45:43 DEBUG 00:45:43 DEBUG 00:45:43 DEBUG 00:45:43 DEBUG 00:45:43 DEBUG 00:45:43 DEBUG 00:45:43 DEBUG 00:45:43 DEBUG [...] 00:45:43 DEBUG "instance": "10.142.0.43:9100" 00:45:43 DEBUG [...] 00:45:43 DEBUG 00:45:43 DEBUG }, 00:45:43 DEBUG 00:45:43 DEBUG 00:45:43 DEBUG 00:45:43 DEBUG "value": [ 00:45:43 DEBUG 00:45:43 DEBUG "pod": "ds-cts-2" 00:45:43 DEBUG 1713483943, 00:45:43 DEBUG 00:45:43 DEBUG }, 00:45:43 DEBUG "0.04199999999999875" 00:45:43 DEBUG "instance": "10.142.0.43:9100" 00:45:43 DEBUG "value": [ 00:45:43 DEBUG ] 00:45:43 DEBUG }, 00:45:43 DEBUG 1713483943, 00:45:43 DEBUG } 00:45:43 DEBUG "value": [ 00:45:43 DEBUG "37895434.74411192" 00:45:43 DEBUG ] 00:45:43 DEBUG 1713483943, 00:45:43 DEBUG ] 00:45:43 DEBUG } 00:45:43 DEBUG "277401660416" 00:45:43 DEBUG } 00:45:43 DEBUG } 00:45:43 DEBUG ] 00:45:43 DEBUG ] 00:45:43 DEBUG } 00:45:43 DEBUG } 00:45:43 DEBUG ] 00:45:43 DEBUG } 00:45:43 DEBUG } 00:45:43 DEBUG } 00:45:43 INFO 00:45:43 INFO [loop_until]: kubectl --namespace=xlou top pods 00:45:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:45:43 INFO 00:45:43 INFO [loop_until]: kubectl --namespace=xlou top node 00:45:43 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713483943" 00:45:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713483943" 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [loop_until]: OK (rc = 0) 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG --- status code --- 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG --- stdout --- 00:45:43 DEBUG --- status code --- 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 DEBUG --- status code --- 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 11258m 7795Mi am-6fd554f8f7-hvvkw 9517m 5929Mi am-6fd554f8f7-pmj4b 9676m 5080Mi ds-cts-0 6910m 11322Mi ds-cts-1 8371m 10973Mi ds-cts-2 8319m 11297Mi ds-idrepo-0 4859m 3257Mi ds-idrepo-1 4870m 3016Mi ds-idrepo-2 4589m 3022Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6962m 886Mi 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG --- http response --- 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG --- stderr --- 00:45:43 INFO [http_cmd]: http status code OK 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 INFO [loop_until]: OK (rc = 0) 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:43 DEBUG 00:45:43 DEBUG --- status code --- 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG --- stdout --- 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:45:43 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5932Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4728m 15% 6671Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9263m 31% 8613Mi 7% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3215Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 6979m 23% 13814Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 136m 0% 5179Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11224m 37% 10554Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 27881m 93% 3128Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10554m 35% 7300Mi 6% gke-xlou-cdm-ds-0f601751-j488 5055m 16% 6058Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 7297m 24% 14917Mi 13% gke-xlou-cdm-ds-0f601751-nj6q 5425m 18% 6622Mi 5% gke-xlou-cdm-ds-0f601751-nmxm 8933m 29% 13286Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 7113m 44% 4657Mi 7% 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG { 00:45:43 DEBUG { 00:45:43 DEBUG { 00:45:43 DEBUG { 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG { 00:45:43 DEBUG { 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG { 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG { 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:45:43 DEBUG { 00:45:43 DEBUG { 00:45:43 DEBUG --- http response --- 00:45:43 DEBUG { 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG --- stderr --- 00:45:43 DEBUG { 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG { 00:45:43 DEBUG { 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG { 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG { 00:45:43 DEBUG "data": { 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:45:43 DEBUG { 00:45:43 DEBUG 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG { 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "data": { 00:45:43 DEBUG "data": { 00:45:43 DEBUG "data": { 00:45:43 DEBUG "data": { 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "data": { 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "data": { 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG "data": { 00:45:43 DEBUG { 00:45:43 DEBUG "data": { 00:45:43 DEBUG "data": { 00:45:43 DEBUG "data": { 00:45:43 DEBUG ----- output ----- 00:45:43 DEBUG { 00:45:43 DEBUG "data": { 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "data": { 00:45:43 DEBUG "data": { 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "data": { 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "data": { 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG { 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "data": { 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG { 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "status": "success", 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "data": { 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "data": { 00:45:43 DEBUG "data": { 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "result": [ 00:45:43 DEBUG { 00:45:43 DEBUG { 00:45:43 DEBUG { 00:45:43 DEBUG { 00:45:43 DEBUG "result": [ 00:45:43 DEBUG { 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "metric": { 00:45:43 DEBUG { 00:45:43 DEBUG "data": { 00:45:43 DEBUG { 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG { 00:45:43 DEBUG { 00:45:43 DEBUG { 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG { 00:45:43 DEBUG "metric": { 00:45:43 DEBUG { 00:45:43 DEBUG { 00:45:43 DEBUG "metric": { 00:45:43 DEBUG "metric": { 00:45:43 DEBUG "metric": { 00:45:43 DEBUG "metric": { 00:45:43 DEBUG { 00:45:43 DEBUG "metric": { 00:45:43 DEBUG { 00:45:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:45:43 DEBUG "metric": { 00:45:43 DEBUG "resultType": "vector", 00:45:43 DEBUG "metric": { 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "metric": { 00:45:43 DEBUG "metric": { 00:45:43 DEBUG "metric": { 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "result": [ 00:45:43 DEBUG "pod": "ds-cts-0" 00:45:43 DEBUG "metric": { 00:45:43 DEBUG "metric": { 00:45:43 DEBUG "pod": "ds-cts-0" 00:45:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:45:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:45:43 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:45:43 DEBUG "metric": { 00:45:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:45:44 DEBUG "metric": { 00:45:44 DEBUG }, 00:45:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:45:44 DEBUG "result": [ 00:45:44 DEBUG "pod": "ds-idrepo-0" 00:45:44 DEBUG }, 00:45:44 DEBUG "pod": "ds-idrepo-0" 00:45:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:45:44 DEBUG "pod": "ds-cts-0" 00:45:44 DEBUG { 00:45:44 DEBUG { 00:45:44 DEBUG { 00:45:44 DEBUG }, 00:45:44 DEBUG "pod": "ds-idrepo-0" 00:45:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:45:44 DEBUG }, 00:45:44 DEBUG }, 00:45:44 DEBUG }, 00:45:44 DEBUG }, 00:45:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:45:44 DEBUG }, 00:45:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:45:44 DEBUG "value": [ 00:45:44 DEBUG }, 00:45:44 DEBUG { 00:45:44 DEBUG { 00:45:44 DEBUG "value": [ 00:45:44 DEBUG }, 00:45:44 DEBUG }, 00:45:44 DEBUG }, 00:45:44 DEBUG "metric": { 00:45:44 DEBUG "metric": { 00:45:44 DEBUG "metric": { 00:45:44 DEBUG "value": [ 00:45:44 DEBUG }, 00:45:44 DEBUG }, 00:45:44 DEBUG "value": [ 00:45:44 DEBUG 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "value": [ 00:45:44 DEBUG }, 00:45:44 DEBUG "value": [ 00:45:44 DEBUG }, 00:45:44 DEBUG 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "metric": { 00:45:44 DEBUG "metric": { 00:45:44 DEBUG 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "pod": "ds-idrepo-0" 00:45:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:45:44 DEBUG "instance": "10.142.0.107:9100" 00:45:44 DEBUG 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "value": [ 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG "value": [ 00:45:44 DEBUG 00:45:44 DEBUG "value": [ 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG "instance": "10.142.0.107:9100" 00:45:44 DEBUG "instance": "10.142.0.107:9100" 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG }, 00:45:44 DEBUG }, 00:45:44 DEBUG }, 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG }, 00:45:44 DEBUG }, 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "value": [ 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG [...] 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "value": [ 00:45:44 DEBUG [...] 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG [...] 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG [...] 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG [...] 00:45:44 DEBUG [...] 00:45:44 DEBUG 00:45:44 DEBUG [...] 00:45:44 DEBUG [...] 00:45:44 DEBUG [...] 00:45:44 DEBUG [...] 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG [...] 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG [...] 00:45:44 DEBUG [...] 00:45:44 DEBUG [...] 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG [...] 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG [...] 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG [...] 00:45:44 DEBUG [...] 00:45:44 DEBUG [...] 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:45:44 DEBUG [...] 00:45:44 DEBUG [...] 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG "pod": "ds-idrepo-2" 00:45:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:45:44 DEBUG }, 00:45:44 DEBUG "value": [ 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG "pod": "ds-cts-2" 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG "pod": "ds-cts-2" 00:45:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:45:44 DEBUG "pod": "ds-cts-2" 00:45:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:45:44 DEBUG 00:45:44 DEBUG "pod": "ds-cts-2" 00:45:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:45:44 DEBUG 00:45:44 DEBUG }, 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG "pod": "ds-idrepo-2" 00:45:44 DEBUG 00:45:44 DEBUG }, 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG "0" 00:45:44 DEBUG }, 00:45:44 DEBUG "pod": "ds-idrepo-2" 00:45:44 DEBUG "pod": "ds-cts-2" 00:45:44 DEBUG }, 00:45:44 DEBUG }, 00:45:44 DEBUG }, 00:45:44 DEBUG }, 00:45:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:45:44 DEBUG }, 00:45:44 DEBUG }, 00:45:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:45:44 DEBUG "value": [ 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG }, 00:45:44 DEBUG 00:45:44 DEBUG "value": [ 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG ] 00:45:44 DEBUG "value": [ 00:45:44 DEBUG }, 00:45:44 DEBUG }, 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "value": [ 00:45:44 DEBUG }, 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "value": [ 00:45:44 DEBUG }, 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "instance": "10.142.0.43:9100" 00:45:44 DEBUG 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "pod": "ds-cts-2" 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG 00:45:44 DEBUG } 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "value": [ 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG "value": [ 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG }, 00:45:44 DEBUG "instance": "10.142.0.43:9100" 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG }, 00:45:44 DEBUG "0" 00:45:44 DEBUG "pod": "ds-cts-2" 00:45:44 DEBUG }, 00:45:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:45:44 DEBUG ] 00:45:44 DEBUG "0" 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG "0" 00:45:44 DEBUG "232357.5" 00:45:44 DEBUG "44929645.61714184" 00:45:44 DEBUG "7.0365086000724935" 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG "0" 00:45:44 DEBUG "1892.7666666666667" 00:45:44 DEBUG "1892.8666666666666" 00:45:44 DEBUG "1892.5333333333333" 00:45:44 DEBUG "value": [ 00:45:44 DEBUG }, 00:45:44 DEBUG "0" 00:45:44 DEBUG "value": [ 00:45:44 DEBUG ] 00:45:44 DEBUG "instance": "10.142.0.43:9100" 00:45:44 DEBUG "value": [ 00:45:44 DEBUG }, 00:45:44 DEBUG "value": [ 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG ] 00:45:44 DEBUG "9.136275615371112" 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG "8.040000000000001" 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG "value": [ 00:45:44 DEBUG ] 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG } 00:45:44 DEBUG }, 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG } 00:45:44 DEBUG "0" 00:45:44 DEBUG "1014616064" 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG "50101113.5" 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG "0.00002766" 00:45:44 DEBUG ] 00:45:44 DEBUG "value": [ 00:45:44 DEBUG "0" 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG 1713483943, 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG "0.02899999999999954" 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG "50105312.36666666" 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG ] 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:45:44 DEBUG } 00:46:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484003" 00:46:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484003" 00:46:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484003" 00:46:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484003" 00:46:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484003" 00:46:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484003" 00:46:43 ERROR [http_cmd]: ERROR 00:46:43 ERROR --- status code --- 00:46:43 INFO [http_cmd]: http status code OK 00:46:43 INFO [http_cmd]: http status code OK 00:46:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:46:43 DEBUG --- status code --- 00:46:43 DEBUG --- status code --- 00:46:43 ERROR --- http response --- 00:46:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:46:43 DEBUG --- http response --- 00:46:43 DEBUG --- http response --- 00:46:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:46:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:46:43 DEBUG ----- output ----- 00:46:43 DEBUG { 00:46:43 DEBUG "status": "success", 00:46:43 DEBUG "data": { 00:46:43 DEBUG "resultType": "vector", 00:46:43 DEBUG "result": [ 00:46:43 DEBUG { 00:46:43 DEBUG "metric": { 00:46:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:46:43 DEBUG }, 00:46:43 DEBUG "value": [ 00:46:43 DEBUG 00:46:43 DEBUG 00:46:43 DEBUG 00:46:43 DEBUG [...] 00:46:43 DEBUG 00:46:43 DEBUG 00:46:43 DEBUG 00:46:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:46:43 DEBUG }, 00:46:43 DEBUG "value": [ 00:46:43 DEBUG 1713484003, 00:46:43 DEBUG "381" 00:46:43 DEBUG ] 00:46:43 DEBUG } 00:46:43 DEBUG ] 00:46:43 DEBUG } 00:46:43 DEBUG } 00:46:43 INFO [http_cmd]: http status code OK 00:46:43 INFO [http_cmd]: http status code OK 00:46:43 DEBUG --- status code --- 00:46:43 DEBUG --- status code --- 00:46:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:43 DEBUG --- http response --- 00:46:43 DEBUG --- http response --- 00:46:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:43 DEBUG ----- output ----- 00:46:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:46:43 INFO [http_cmd]: http status code OK 00:46:43 DEBUG { 00:46:43 DEBUG ----- output ----- 00:46:43 DEBUG --- status code --- 00:46:43 DEBUG "status": "success", 00:46:43 DEBUG { 00:46:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:43 DEBUG "data": { 00:46:43 DEBUG "status": "success", 00:46:43 DEBUG --- http response --- 00:46:43 DEBUG "resultType": "vector", 00:46:43 DEBUG "data": { 00:46:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:43 DEBUG "result": [ 00:46:43 DEBUG "resultType": "vector", 00:46:43 DEBUG ----- output ----- 00:46:43 DEBUG { 00:46:43 DEBUG "result": [ 00:46:43 DEBUG { 00:46:43 DEBUG "metric": { 00:46:43 DEBUG { 00:46:43 DEBUG "status": "success", 00:46:43 DEBUG "instance": "10.142.0.107:9100" 00:46:43 DEBUG "metric": { 00:46:43 DEBUG "data": { 00:46:43 DEBUG }, 00:46:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:46:43 DEBUG "resultType": "vector", 00:46:43 DEBUG "value": [ 00:46:43 DEBUG }, 00:46:43 DEBUG "result": [ 00:46:43 DEBUG 00:46:43 DEBUG "value": [ 00:46:43 DEBUG { 00:46:43 DEBUG 00:46:43 DEBUG 00:46:43 DEBUG "metric": { 00:46:43 DEBUG 00:46:43 DEBUG 00:46:43 DEBUG "instance": "10.142.0.107:9100" 00:46:43 DEBUG [...] 00:46:43 DEBUG 00:46:43 DEBUG }, 00:46:43 DEBUG 00:46:43 DEBUG [...] 00:46:43 DEBUG "value": [ 00:46:43 DEBUG 00:46:43 DEBUG 00:46:43 DEBUG 00:46:43 DEBUG 00:46:43 DEBUG 00:46:43 DEBUG 00:46:43 DEBUG "instance": "10.142.0.43:9100" 00:46:43 DEBUG 00:46:43 DEBUG 00:46:43 DEBUG }, 00:46:43 DEBUG "pod": "ds-idrepo-0" 00:46:43 DEBUG [...] 00:46:43 DEBUG "value": [ 00:46:43 DEBUG }, 00:46:43 DEBUG 00:46:43 DEBUG 1713484003, 00:46:43 DEBUG "value": [ 00:46:43 DEBUG 00:46:43 DEBUG "0.041666666666667615" 00:46:43 DEBUG 1713484003, 00:46:43 DEBUG 00:46:43 DEBUG ] 00:46:43 DEBUG "5096.270490528476" 00:46:43 DEBUG "instance": "10.142.0.43:9100" 00:46:43 DEBUG } 00:46:43 DEBUG ] 00:46:43 DEBUG }, 00:46:43 DEBUG ] 00:46:43 DEBUG } 00:46:43 DEBUG "value": [ 00:46:43 DEBUG } 00:46:43 DEBUG ] 00:46:43 DEBUG 1713484003, 00:46:43 DEBUG } 00:46:43 DEBUG } 00:46:43 DEBUG "279719625728" 00:46:43 DEBUG } 00:46:43 DEBUG ] 00:46:43 DEBUG } 00:46:43 DEBUG ] 00:46:43 DEBUG } 00:46:43 DEBUG } 00:46:44 INFO 00:46:44 INFO [loop_until]: kubectl --namespace=xlou top pods 00:46:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:46:44 INFO 00:46:44 INFO [loop_until]: kubectl --namespace=xlou top node 00:46:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484004" 00:46:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484004" 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [loop_until]: OK (rc = 0) 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG --- status code --- 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 DEBUG --- status code --- 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG --- status code --- 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 DEBUG --- status code --- 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 DEBUG --- status code --- 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG --- stdout --- 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10606m 7904Mi am-6fd554f8f7-hvvkw 8868m 8816Mi am-6fd554f8f7-pmj4b 10766m 8285Mi ds-cts-0 7203m 11956Mi ds-cts-1 9249m 11950Mi ds-cts-2 8408m 12013Mi ds-idrepo-0 4897m 3257Mi ds-idrepo-1 4863m 3087Mi ds-idrepo-2 4582m 3022Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 4m 1215Mi lodemon-5d46798758-5h2st 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6837m 888Mi 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG --- stderr --- 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:44 INFO [http_cmd]: http status code OK 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:44 DEBUG 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG --- status code --- 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG { 00:46:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG { 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG --- http response --- 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG { 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "data": { 00:46:44 DEBUG "data": { 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "data": { 00:46:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "data": { 00:46:44 DEBUG "data": { 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "data": { 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "data": { 00:46:44 DEBUG "data": { 00:46:44 DEBUG "data": { 00:46:44 DEBUG "data": { 00:46:44 DEBUG "data": { 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "data": { 00:46:44 DEBUG "data": { 00:46:44 DEBUG "data": { 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG ----- output ----- 00:46:44 DEBUG "data": { 00:46:44 DEBUG "data": { 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "data": { 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "data": { 00:46:44 DEBUG "data": { 00:46:44 DEBUG "data": { 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "result": [ 00:46:44 DEBUG "result": [ 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "result": [ 00:46:44 DEBUG { 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "result": [ 00:46:44 DEBUG "result": [ 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "result": [ 00:46:44 DEBUG { 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "result": [ 00:46:44 DEBUG "result": [ 00:46:44 DEBUG "result": [ 00:46:44 DEBUG "result": [ 00:46:44 DEBUG "result": [ 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG "result": [ 00:46:44 DEBUG "result": [ 00:46:44 DEBUG "result": [ 00:46:44 DEBUG { 00:46:44 DEBUG "status": "success", 00:46:44 DEBUG "result": [ 00:46:44 DEBUG "result": [ 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG "result": [ 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "result": [ 00:46:44 DEBUG "result": [ 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "metric": { 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "data": { 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:46:44 DEBUG { 00:46:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:46:44 DEBUG { 00:46:44 DEBUG { 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:46:44 DEBUG "pod": "ds-cts-0" 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:46:44 DEBUG "resultType": "vector", 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "pod": "ds-idrepo-0" 00:46:44 DEBUG "result": [ 00:46:44 DEBUG }, 00:46:44 DEBUG "metric": { 00:46:44 DEBUG }, 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "pod": "ds-cts-0" 00:46:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:46:44 DEBUG "instance": "10.142.0.107:9100" 00:46:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:46:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:46:44 DEBUG }, 00:46:44 DEBUG }, 00:46:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:46:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:46:44 DEBUG "pod": "ds-idrepo-0" 00:46:44 DEBUG }, 00:46:44 DEBUG "result": [ 00:46:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:46:44 DEBUG "pod": "ds-idrepo-0" 00:46:44 DEBUG }, 00:46:44 DEBUG { 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "pod": "ds-idrepo-0" 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "pod": "ds-cts-0" 00:46:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:46:44 DEBUG }, 00:46:44 DEBUG }, 00:46:44 DEBUG }, 00:46:44 DEBUG }, 00:46:44 DEBUG }, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "value": [ 00:46:44 DEBUG }, 00:46:44 DEBUG }, 00:46:44 DEBUG }, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG { 00:46:44 DEBUG }, 00:46:44 DEBUG }, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "metric": { 00:46:44 DEBUG 00:46:44 DEBUG }, 00:46:44 DEBUG 00:46:44 DEBUG }, 00:46:44 DEBUG }, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "value": [ 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "value": [ 00:46:44 DEBUG 00:46:44 DEBUG "metric": { 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "value": [ 00:46:44 DEBUG 00:46:44 DEBUG "instance": "10.142.0.107:9100" 00:46:44 DEBUG 00:46:44 DEBUG "value": [ 00:46:44 DEBUG 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "value": [ 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG "instance": "10.142.0.107:9100" 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG }, 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG }, 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG "value": [ 00:46:44 DEBUG [...] 00:46:44 DEBUG 00:46:44 DEBUG [...] 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG [...] 00:46:44 DEBUG 00:46:44 DEBUG [...] 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG [...] 00:46:44 DEBUG "value": [ 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG [...] 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG [...] 00:46:44 DEBUG [...] 00:46:44 DEBUG [...] 00:46:44 DEBUG [...] 00:46:44 DEBUG 00:46:44 DEBUG [...] 00:46:44 DEBUG 00:46:44 DEBUG [...] 00:46:44 DEBUG [...] 00:46:44 DEBUG [...] 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG [...] 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG [...] 00:46:44 DEBUG 00:46:44 DEBUG [...] 00:46:44 DEBUG [...] 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG [...] 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG [...] 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG [...] 00:46:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:46:44 DEBUG 00:46:44 DEBUG "pod": "ds-idrepo-0" 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG "pod": "ds-idrepo-0" 00:46:44 DEBUG 00:46:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:46:44 DEBUG 00:46:44 DEBUG "pod": "ds-cts-2" 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG "pod": "ds-idrepo-2" 00:46:44 DEBUG 00:46:44 DEBUG }, 00:46:44 DEBUG 00:46:44 DEBUG }, 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG "pod": "ds-cts-2" 00:46:44 DEBUG 00:46:44 DEBUG }, 00:46:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:46:44 DEBUG }, 00:46:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:46:44 DEBUG }, 00:46:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:46:44 DEBUG "pod": "ds-idrepo-0" 00:46:44 DEBUG "pod": "ds-cts-2" 00:46:44 DEBUG }, 00:46:44 DEBUG 00:46:44 DEBUG 00:46:44 DEBUG "pod": "ds-idrepo-2" 00:46:44 DEBUG }, 00:46:44 DEBUG 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "pod": "ds-idrepo-2" 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "pod": "ds-cts-2" 00:46:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:46:44 DEBUG }, 00:46:44 DEBUG "instance": "10.142.0.43:9100" 00:46:44 DEBUG "value": [ 00:46:44 DEBUG }, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG }, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG }, 00:46:44 DEBUG }, 00:46:44 DEBUG }, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:46:44 DEBUG 00:46:44 DEBUG }, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG }, 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG }, 00:46:44 DEBUG }, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG }, 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG }, 00:46:44 DEBUG "instance": "10.142.0.43:9100" 00:46:44 DEBUG "value": [ 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG "instance": "10.142.0.43:9100" 00:46:44 DEBUG "6.952630108927769" 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "35291.6731469097" 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "value": [ 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "0" 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG "346788.5" 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG "0" 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "10.6745" 00:46:44 DEBUG ] 00:46:44 DEBUG "0" 00:46:44 DEBUG "value": [ 00:46:44 DEBUG }, 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG "0" 00:46:44 DEBUG ] 00:46:44 DEBUG ] 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG ] 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG "23.666666666666668" 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG ] 00:46:44 DEBUG "0" 00:46:44 DEBUG ] 00:46:44 DEBUG "1900.8666666666666" 00:46:44 DEBUG ] 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG } 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG ] 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG "value": [ 00:46:44 DEBUG "0" 00:46:44 DEBUG }, 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG "0" 00:46:44 DEBUG } 00:46:44 DEBUG "0.000033333333333333335" 00:46:44 DEBUG "1030561792" 00:46:44 DEBUG ] 00:46:44 DEBUG "49932650.86666666" 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG "4.871640579800228" 00:46:44 DEBUG ] 00:46:44 DEBUG "1899.7333333333333" 00:46:44 DEBUG } 00:46:44 DEBUG "1898.7333333333333" 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG ] 00:46:44 DEBUG "value": [ 00:46:44 DEBUG ] 00:46:44 DEBUG ] 00:46:44 DEBUG ] 00:46:44 DEBUG ] 00:46:44 DEBUG ] 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG ] 00:46:44 DEBUG ] 00:46:44 DEBUG "0.029666666666667042" 00:46:44 DEBUG } 00:46:44 DEBUG 1713484004, 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG "49936660.43333334" 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 INFO [loop_until]: OK (rc = 0) 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG } 00:46:44 DEBUG ] 00:46:44 DEBUG --- stdout --- 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5935Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4840m 16% 6656Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8878m 29% 11490Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3206Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 7701m 25% 14375Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5207Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10762m 36% 10676Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 27986m 93% 3140Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10995m 36% 10478Mi 9% gke-xlou-cdm-ds-0f601751-j488 5094m 17% 6068Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8577m 28% 15896Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5425m 18% 6712Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8336m 27% 13875Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7107m 44% 4656Mi 7% 00:46:44 DEBUG } 00:46:44 DEBUG } 00:46:44 DEBUG --- stderr --- 00:46:44 DEBUG 00:47:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484063" 00:47:43 ERROR [http_cmd]: ERROR 00:47:43 ERROR --- status code --- 00:47:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:47:43 ERROR --- http response --- 00:47:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:47:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:47:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484063" 00:47:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484063" 00:47:43 INFO [http_cmd]: http status code OK 00:47:43 DEBUG --- status code --- 00:47:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:43 DEBUG --- http response --- 00:47:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484063" 00:47:43 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:47:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484063" 00:47:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484063" 00:47:43 INFO [http_cmd]: http status code OK 00:47:43 DEBUG --- status code --- 00:47:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:43 DEBUG --- http response --- 00:47:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:43 DEBUG ----- output ----- 00:47:43 DEBUG { 00:47:43 DEBUG "status": "success", 00:47:43 DEBUG "data": { 00:47:43 DEBUG "resultType": "vector", 00:47:43 DEBUG "result": [ 00:47:43 DEBUG { 00:47:43 DEBUG "metric": { 00:47:43 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:47:43 DEBUG }, 00:47:43 DEBUG "value": [ 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG [...] 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:47:43 DEBUG }, 00:47:43 DEBUG "value": [ 00:47:43 DEBUG 1713484063, 00:47:43 DEBUG "666.6666666666666" 00:47:43 DEBUG ] 00:47:43 DEBUG } 00:47:43 DEBUG ] 00:47:43 DEBUG } 00:47:43 DEBUG } 00:47:43 INFO [http_cmd]: http status code OK 00:47:43 DEBUG --- status code --- 00:47:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:43 DEBUG --- http response --- 00:47:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:47:43 DEBUG ----- output ----- 00:47:43 DEBUG { 00:47:43 DEBUG "status": "success", 00:47:43 DEBUG "data": { 00:47:43 DEBUG "resultType": "vector", 00:47:43 DEBUG "result": [ 00:47:43 DEBUG { 00:47:43 DEBUG "metric": { 00:47:43 DEBUG "instance": "10.142.0.107:9100" 00:47:43 DEBUG }, 00:47:43 DEBUG "value": [ 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG [...] 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG "instance": "10.142.0.43:9100" 00:47:43 DEBUG }, 00:47:43 DEBUG "value": [ 00:47:43 DEBUG 1713484063, 00:47:43 DEBUG "0.04390000000000119" 00:47:43 DEBUG ] 00:47:43 DEBUG } 00:47:43 DEBUG ] 00:47:43 DEBUG } 00:47:43 DEBUG } 00:47:43 INFO [http_cmd]: http status code OK 00:47:43 DEBUG --- status code --- 00:47:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:43 DEBUG --- http response --- 00:47:43 INFO [http_cmd]: http status code OK 00:47:43 DEBUG --- status code --- 00:47:43 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:43 DEBUG --- http response --- 00:47:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:47:43 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:47:43 DEBUG ----- output ----- 00:47:43 DEBUG ----- output ----- 00:47:43 DEBUG { 00:47:43 DEBUG { 00:47:43 DEBUG "status": "success", 00:47:43 DEBUG "status": "success", 00:47:43 DEBUG "data": { 00:47:43 DEBUG "data": { 00:47:43 DEBUG "resultType": "vector", 00:47:43 DEBUG "resultType": "vector", 00:47:43 DEBUG "result": [ 00:47:43 DEBUG "result": [ 00:47:43 DEBUG { 00:47:43 DEBUG { 00:47:43 DEBUG "metric": { 00:47:43 DEBUG "metric": { 00:47:43 DEBUG "instance": "10.142.0.107:9100" 00:47:43 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:47:43 DEBUG }, 00:47:43 DEBUG }, 00:47:43 DEBUG "value": [ 00:47:43 DEBUG "value": [ 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG [...] 00:47:43 DEBUG [...] 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG 00:47:43 DEBUG "instance": "10.142.0.43:9100" 00:47:43 DEBUG "pod": "ds-idrepo-0" 00:47:43 DEBUG }, 00:47:43 DEBUG }, 00:47:43 DEBUG "value": [ 00:47:43 DEBUG "value": [ 00:47:43 DEBUG 1713484063, 00:47:43 DEBUG 1713484063, 00:47:43 DEBUG "282002285568" 00:47:43 DEBUG "3272.5981818835107" 00:47:43 DEBUG ] 00:47:43 DEBUG ] 00:47:43 DEBUG } 00:47:43 DEBUG } 00:47:43 DEBUG ] 00:47:43 DEBUG ] 00:47:43 DEBUG } 00:47:43 DEBUG } 00:47:43 DEBUG } 00:47:43 DEBUG } 00:47:44 INFO 00:47:44 INFO [loop_until]: kubectl --namespace=xlou top pods 00:47:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG { 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "data": { 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "result": [ 00:47:44 DEBUG { 00:47:44 DEBUG "metric": { 00:47:44 DEBUG "pod": "ds-idrepo-0" 00:47:44 DEBUG }, 00:47:44 DEBUG "value": [ 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG [...] 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG "pod": "ds-cts-2" 00:47:44 DEBUG }, 00:47:44 DEBUG "value": [ 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG "0" 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484064" 00:47:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484064" 00:47:44 INFO 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 INFO [loop_until]: OK (rc = 0) 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 INFO [loop_until]: kubectl --namespace=xlou top node 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG --- status code --- 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 DEBUG --- status code --- 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 DEBUG --- stdout --- 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG --- status code --- 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 DEBUG --- status code --- 00:47:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 11123m 7935Mi am-6fd554f8f7-hvvkw 8651m 8789Mi am-6fd554f8f7-pmj4b 9346m 8261Mi ds-cts-0 7354m 11963Mi ds-cts-1 8926m 11958Mi ds-cts-2 8492m 12019Mi ds-idrepo-0 4901m 3257Mi ds-idrepo-1 4675m 3095Mi ds-idrepo-2 4663m 3023Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6706m 898Mi 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG --- status code --- 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG --- stderr --- 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG --- http response --- 00:47:44 INFO [http_cmd]: http status code OK 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:47:44 DEBUG --- status code --- 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG { 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG { 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG { 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "data": { 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG { 00:47:44 DEBUG --- http response --- 00:47:44 DEBUG "data": { 00:47:44 DEBUG "data": { 00:47:44 DEBUG "data": { 00:47:44 DEBUG { 00:47:44 DEBUG "data": { 00:47:44 DEBUG "data": { 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG { 00:47:44 DEBUG "data": { 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "data": { 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "data": { 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "data": { 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "data": { 00:47:44 DEBUG "data": { 00:47:44 DEBUG "data": { 00:47:44 DEBUG "data": { 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "data": { 00:47:44 DEBUG ----- output ----- 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "result": [ 00:47:44 DEBUG { 00:47:44 DEBUG "data": { 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "data": { 00:47:44 DEBUG "data": { 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "data": { 00:47:44 DEBUG { 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "result": [ 00:47:44 DEBUG { 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "metric": { 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "result": [ 00:47:44 DEBUG { 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "result": [ 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "metric": { 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG "metric": { 00:47:44 DEBUG { 00:47:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "metric": { 00:47:44 DEBUG "metric": { 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG { 00:47:44 DEBUG "metric": { 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG { 00:47:44 DEBUG "metric": { 00:47:44 DEBUG "metric": { 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "pod": "ds-cts-0" 00:47:44 DEBUG "metric": { 00:47:44 DEBUG "status": "success", 00:47:44 DEBUG "metric": { 00:47:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:47:44 DEBUG "metric": { 00:47:44 DEBUG }, 00:47:44 DEBUG { 00:47:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:47:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:47:44 DEBUG "result": [ 00:47:44 DEBUG "metric": { 00:47:44 DEBUG "pod": "ds-idrepo-0" 00:47:44 DEBUG "metric": { 00:47:44 DEBUG "metric": { 00:47:44 DEBUG "metric": { 00:47:44 DEBUG "metric": { 00:47:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:47:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:47:44 DEBUG { 00:47:44 DEBUG }, 00:47:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:47:44 DEBUG "data": { 00:47:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:47:44 DEBUG }, 00:47:44 DEBUG "pod": "ds-idrepo-0" 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "metric": { 00:47:44 DEBUG }, 00:47:44 DEBUG }, 00:47:44 DEBUG { 00:47:44 DEBUG "instance": "10.142.0.107:9100" 00:47:44 DEBUG }, 00:47:44 DEBUG "pod": "ds-cts-0" 00:47:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:47:44 DEBUG "pod": "ds-cts-0" 00:47:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:47:44 DEBUG }, 00:47:44 DEBUG }, 00:47:44 DEBUG "metric": { 00:47:44 DEBUG "pod": "ds-idrepo-0" 00:47:44 DEBUG }, 00:47:44 DEBUG }, 00:47:44 DEBUG }, 00:47:44 DEBUG "value": [ 00:47:44 DEBUG }, 00:47:44 DEBUG 00:47:44 DEBUG "instance": "10.142.0.107:9100" 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "metric": { 00:47:44 DEBUG }, 00:47:44 DEBUG "value": [ 00:47:44 DEBUG }, 00:47:44 DEBUG }, 00:47:44 DEBUG }, 00:47:44 DEBUG }, 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "resultType": "vector", 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "value": [ 00:47:44 DEBUG 00:47:44 DEBUG "value": [ 00:47:44 DEBUG 00:47:44 DEBUG }, 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG "instance": "10.142.0.107:9100" 00:47:44 DEBUG "value": [ 00:47:44 DEBUG 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "value": [ 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG "value": [ 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG }, 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG "result": [ 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG [...] 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG "value": [ 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG [...] 00:47:44 DEBUG { 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG [...] 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG [...] 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG [...] 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG [...] 00:47:44 DEBUG [...] 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG [...] 00:47:44 DEBUG [...] 00:47:44 DEBUG [...] 00:47:44 DEBUG 00:47:44 DEBUG [...] 00:47:44 DEBUG 00:47:44 DEBUG [...] 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG [...] 00:47:44 DEBUG 00:47:44 DEBUG [...] 00:47:44 DEBUG [...] 00:47:44 DEBUG [...] 00:47:44 DEBUG [...] 00:47:44 DEBUG [...] 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG "metric": { 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:47:44 DEBUG 00:47:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG [...] 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG }, 00:47:44 DEBUG 00:47:44 DEBUG }, 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:47:44 DEBUG 00:47:44 DEBUG }, 00:47:44 DEBUG 00:47:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG "pod": "ds-idrepo-2" 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG 00:47:44 DEBUG "pod": "ds-idrepo-0" 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "pod": "ds-cts-2" 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "pod": "ds-idrepo-2" 00:47:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:47:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:47:44 DEBUG }, 00:47:44 DEBUG "pod": "ds-idrepo-2" 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "instance": "10.142.0.43:9100" 00:47:44 DEBUG }, 00:47:44 DEBUG 00:47:44 DEBUG "instance": "10.142.0.43:9100" 00:47:44 DEBUG }, 00:47:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:47:44 DEBUG "pod": "ds-cts-2" 00:47:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:47:44 DEBUG "pod": "ds-cts-2" 00:47:44 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:47:44 DEBUG }, 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG }, 00:47:44 DEBUG 00:47:44 DEBUG }, 00:47:44 DEBUG }, 00:47:44 DEBUG }, 00:47:44 DEBUG "value": [ 00:47:44 DEBUG }, 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG }, 00:47:44 DEBUG "value": [ 00:47:44 DEBUG 00:47:44 DEBUG }, 00:47:44 DEBUG "value": [ 00:47:44 DEBUG }, 00:47:44 DEBUG }, 00:47:44 DEBUG }, 00:47:44 DEBUG }, 00:47:44 DEBUG }, 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "12.77" 00:47:44 DEBUG "value": [ 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG "4.1" 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "value": [ 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "0" 00:47:44 DEBUG "value": [ 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG "instance": "10.142.0.43:9100" 00:47:44 DEBUG "value": [ 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "value": [ 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG ] 00:47:44 DEBUG 00:47:44 DEBUG "value": [ 00:47:44 DEBUG ] 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG "1909.9333333333334" 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG ] 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG "461283.5" 00:47:44 DEBUG }, 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG "0" 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG "0" 00:47:44 DEBUG } 00:47:44 DEBUG 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG } 00:47:44 DEBUG "1909.8666666666666" 00:47:44 DEBUG "1033105408" 00:47:44 DEBUG ] 00:47:44 DEBUG "0" 00:47:44 DEBUG } 00:47:44 DEBUG "49850473.63333333" 00:47:44 DEBUG ] 00:47:44 DEBUG "value": [ 00:47:44 DEBUG "0.030333333333333583" 00:47:44 DEBUG ] 00:47:44 DEBUG "1909.8999999999999" 00:47:44 DEBUG "0" 00:47:44 DEBUG "0" 00:47:44 DEBUG "0.000056431111111111105" 00:47:44 DEBUG "0" 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG [...] 00:47:44 DEBUG "0" 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG ] 00:47:44 DEBUG "49845741.63333333" 00:47:44 DEBUG } 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG }, 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 DEBUG "value": [ 00:47:44 DEBUG } 00:47:44 DEBUG 1713484064, 00:47:44 DEBUG "7.0276835085983675" 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG ] 00:47:44 DEBUG } 00:47:44 DEBUG } 00:47:44 INFO [loop_until]: OK (rc = 0) 00:47:44 DEBUG --- stdout --- 00:47:44 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5929Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4808m 16% 6660Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8849m 29% 11519Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3212Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 7401m 24% 14377Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 168m 0% 5202Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11307m 37% 10721Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 27862m 93% 3129Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10259m 34% 10466Mi 9% gke-xlou-cdm-ds-0f601751-j488 5064m 16% 6067Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9103m 30% 15917Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5337m 17% 6736Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8138m 27% 13880Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7112m 44% 4666Mi 7% 00:47:44 DEBUG --- stderr --- 00:47:44 DEBUG 00:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484123" 00:48:43 ERROR [http_cmd]: ERROR 00:48:43 ERROR --- status code --- 00:48:43 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:48:43 ERROR --- http response --- 00:48:43 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:48:43 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484123" 00:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484123" 00:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484123" 00:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484123" 00:48:43 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484123" 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG { 00:48:44 DEBUG { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "data": { 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "result": [ 00:48:44 DEBUG "result": [ 00:48:44 DEBUG { 00:48:44 DEBUG { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:48:44 DEBUG "instance": "10.142.0.107:9100" 00:48:44 DEBUG }, 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:48:44 DEBUG "instance": "10.142.0.43:9100" 00:48:44 DEBUG }, 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 1713484123, 00:48:44 DEBUG 1713484123, 00:48:44 DEBUG "0.04213333333333367" 00:48:44 DEBUG "500" 00:48:44 DEBUG ] 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "result": [ 00:48:44 DEBUG { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "instance": "10.142.0.107:9100" 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG "instance": "10.142.0.43:9100" 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 1713484123, 00:48:44 DEBUG "284267138048" 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "result": [ 00:48:44 DEBUG { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 1713484123, 00:48:44 DEBUG "0" 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "result": [ 00:48:44 DEBUG { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "pod": "ds-idrepo-0" 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG "pod": "ds-cts-2" 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG "0" 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 INFO 00:48:44 INFO [loop_until]: kubectl --namespace=xlou top pods 00:48:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484124" 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "result": [ 00:48:44 DEBUG { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG "2243189.900447533" 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "result": [ 00:48:44 DEBUG { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "pod": "ds-cts-0" 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG "pod": "ds-cts-2" 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG "0" 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "result": [ 00:48:44 DEBUG { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "instance": "10.142.0.107:9100" 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG "instance": "10.142.0.43:9100" 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG "49736263.53333333" 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484124" 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484124" 00:48:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484124" 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 DEBUG --- status code --- 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 INFO [loop_until]: OK (rc = 0) 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 INFO 00:48:44 DEBUG --- status code --- 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 DEBUG --- status code --- 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG --- stdout --- 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG --- status code --- 00:48:44 INFO [loop_until]: kubectl --namespace=xlou top node 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG --- status code --- 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10409m 9378Mi am-6fd554f8f7-hvvkw 8861m 8862Mi am-6fd554f8f7-pmj4b 9614m 8250Mi ds-cts-0 7196m 11970Mi ds-cts-1 9793m 11965Mi ds-cts-2 8758m 12026Mi ds-idrepo-0 4831m 3257Mi ds-idrepo-1 4774m 3109Mi ds-idrepo-2 4574m 3037Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 10m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6947m 901Mi 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG --- stderr --- 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:44 DEBUG --- http response --- 00:48:44 INFO [http_cmd]: http status code OK 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:44 DEBUG 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:48:44 DEBUG --- status code --- 00:48:44 DEBUG { 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG { 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG { 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG { 00:48:44 DEBUG { 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG { 00:48:44 DEBUG { 00:48:44 DEBUG { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG { 00:48:44 DEBUG --- http response --- 00:48:44 DEBUG "data": { 00:48:44 DEBUG { 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG { 00:48:44 DEBUG { 00:48:44 DEBUG { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "data": { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "data": { 00:48:44 DEBUG "data": { 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "data": { 00:48:44 DEBUG ----- output ----- 00:48:44 DEBUG "result": [ 00:48:44 DEBUG "data": { 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "data": { 00:48:44 DEBUG "data": { 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "result": [ 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG { 00:48:44 DEBUG { 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "data": { 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "result": [ 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "result": [ 00:48:44 DEBUG "result": [ 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "result": [ 00:48:44 DEBUG "result": [ 00:48:44 DEBUG "result": [ 00:48:44 DEBUG { 00:48:44 DEBUG "result": [ 00:48:44 DEBUG "status": "success", 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "result": [ 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG "result": [ 00:48:44 DEBUG { 00:48:44 DEBUG "result": [ 00:48:44 DEBUG { 00:48:44 DEBUG { 00:48:44 DEBUG "result": [ 00:48:44 DEBUG "result": [ 00:48:44 DEBUG "result": [ 00:48:44 DEBUG { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG { 00:48:44 DEBUG "data": { 00:48:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:48:44 DEBUG { 00:48:44 DEBUG "result": [ 00:48:44 DEBUG { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG { 00:48:44 DEBUG { 00:48:44 DEBUG { 00:48:44 DEBUG "instance": "10.142.0.107:9100" 00:48:44 DEBUG { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "resultType": "vector", 00:48:44 DEBUG }, 00:48:44 DEBUG "metric": { 00:48:44 DEBUG { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "pod": "ds-idrepo-0" 00:48:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "metric": { 00:48:44 DEBUG }, 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "pod": "ds-idrepo-0" 00:48:44 DEBUG }, 00:48:44 DEBUG "pod": "ds-cts-0" 00:48:44 DEBUG "result": [ 00:48:44 DEBUG "value": [ 00:48:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:48:44 DEBUG "metric": { 00:48:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:48:44 DEBUG }, 00:48:44 DEBUG "instance": "10.142.0.107:9100" 00:48:44 DEBUG }, 00:48:44 DEBUG }, 00:48:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:48:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:48:44 DEBUG "pod": "ds-cts-0" 00:48:44 DEBUG "value": [ 00:48:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG }, 00:48:44 DEBUG { 00:48:44 DEBUG 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG "value": [ 00:48:44 DEBUG "value": [ 00:48:44 DEBUG }, 00:48:44 DEBUG }, 00:48:44 DEBUG }, 00:48:44 DEBUG 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG "value": [ 00:48:44 DEBUG "metric": { 00:48:44 DEBUG 00:48:44 DEBUG "pod": "ds-idrepo-0" 00:48:44 DEBUG 00:48:44 DEBUG "value": [ 00:48:44 DEBUG }, 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG "value": [ 00:48:44 DEBUG "value": [ 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:48:44 DEBUG 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG }, 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG [...] 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG 00:48:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG }, 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG "instance": "10.142.0.43:9100" 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:48:44 DEBUG 00:48:44 DEBUG [...] 00:48:44 DEBUG }, 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:48:44 DEBUG 00:48:44 DEBUG "value": [ 00:48:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG "pod": "ds-cts-2" 00:48:44 DEBUG }, 00:48:44 DEBUG "pod": "ds-idrepo-2" 00:48:44 DEBUG }, 00:48:44 DEBUG "pod": "ds-cts-2" 00:48:44 DEBUG 00:48:44 DEBUG "value": [ 00:48:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:48:44 DEBUG 00:48:44 DEBUG 00:48:44 DEBUG }, 00:48:44 DEBUG 00:48:44 DEBUG "instance": "10.142.0.43:9100" 00:48:44 DEBUG }, 00:48:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:48:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:48:44 DEBUG }, 00:48:44 DEBUG 00:48:44 DEBUG "value": [ 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG }, 00:48:44 DEBUG 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG }, 00:48:44 DEBUG "pod": "ds-idrepo-2" 00:48:44 DEBUG "pod": "ds-idrepo-2" 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG }, 00:48:44 DEBUG }, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 00:48:44 DEBUG "1917.9333333333334" 00:48:44 DEBUG "value": [ 00:48:44 DEBUG }, 00:48:44 DEBUG }, 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG "1917.6" 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG "value": [ 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG }, 00:48:44 DEBUG "49741270.233333334" 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG "14.905999999999999" 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:48:44 DEBUG ] 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG "value": [ 00:48:44 DEBUG "value": [ 00:48:44 DEBUG "5.147066235726315" 00:48:44 DEBUG ] 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG "576269.5" 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG "0.0005860955555555556" 00:48:44 DEBUG "value": [ 00:48:44 DEBUG ] 00:48:44 DEBUG "0" 00:48:44 DEBUG ] 00:48:44 DEBUG "0" 00:48:44 DEBUG }, 00:48:44 DEBUG } 00:48:44 DEBUG "1041154048" 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG "0.028333333333333054" 00:48:44 DEBUG ] 00:48:44 DEBUG "0" 00:48:44 DEBUG "1917.8999999999999" 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG "value": [ 00:48:44 DEBUG ] 00:48:44 DEBUG "0" 00:48:44 DEBUG "0" 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG ] 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG ] 00:48:44 DEBUG ] 00:48:44 DEBUG ] 00:48:44 DEBUG ] 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG 1713484124, 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG ] 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG "0" 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG "6.897196943797821" 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG ] 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG ] 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 DEBUG } 00:48:44 INFO [loop_until]: OK (rc = 0) 00:48:44 DEBUG --- stdout --- 00:48:44 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5921Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4634m 15% 6651Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8744m 29% 11580Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3210Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 7294m 24% 14391Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5206Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10502m 35% 12166Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27807m 93% 3139Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10542m 35% 10504Mi 9% gke-xlou-cdm-ds-0f601751-j488 5085m 17% 6084Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9392m 31% 15919Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5353m 17% 6734Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8720m 29% 13878Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7076m 44% 4667Mi 7% 00:48:44 DEBUG --- stderr --- 00:48:44 DEBUG 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484184" 00:49:44 ERROR [http_cmd]: ERROR 00:49:44 ERROR --- status code --- 00:49:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:49:44 ERROR --- http response --- 00:49:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:49:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484184" 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "data": { 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "result": [ 00:49:44 DEBUG { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "instance": "10.142.0.107:9100" 00:49:44 DEBUG }, 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "instance": "10.142.0.43:9100" 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "0.04390000000000024" 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "data": { 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "result": [ 00:49:44 DEBUG { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "500" 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "data": { 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "result": [ 00:49:44 DEBUG { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "instance": "10.142.0.107:9100" 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "instance": "10.142.0.43:9100" 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "286601534464" 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "data": { 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "result": [ 00:49:44 DEBUG { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "0" 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "data": { 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "result": [ 00:49:44 DEBUG { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "pod": "ds-idrepo-0" 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "pod": "ds-cts-2" 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "0" 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484184" 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG --- status code --- 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG { 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG { 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG "data": { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "data": { 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:44 DEBUG "result": [ 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG { 00:49:44 DEBUG "result": [ 00:49:44 DEBUG { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "pod": "ds-cts-0" 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "data": { 00:49:44 DEBUG }, 00:49:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "value": [ 00:49:44 DEBUG }, 00:49:44 DEBUG "result": [ 00:49:44 DEBUG 00:49:44 DEBUG "value": [ 00:49:44 DEBUG { 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "metric": { 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "instance": "10.142.0.107:9100" 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG }, 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "pod": "ds-cts-2" 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG }, 00:49:44 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:49:44 DEBUG [...] 00:49:44 DEBUG "value": [ 00:49:44 DEBUG }, 00:49:44 DEBUG 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG "0" 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG 00:49:44 DEBUG ] 00:49:44 DEBUG "39503.45396187966" 00:49:44 DEBUG "instance": "10.142.0.43:9100" 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG }, 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG "value": [ 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG "49867724.03333333" 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 INFO 00:49:44 INFO [loop_until]: kubectl --namespace=xlou top pods 00:49:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "data": { 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "result": [ 00:49:44 DEBUG { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "pod": "ds-idrepo-0" 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "pod": "ds-idrepo-2" 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "0" 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484184" 00:49:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484184" 00:49:44 INFO [loop_until]: OK (rc = 0) 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG --- stdout --- 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG --- status code --- 00:49:44 INFO 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG --- status code --- 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10692m 9371Mi am-6fd554f8f7-hvvkw 9120m 8864Mi am-6fd554f8f7-pmj4b 9591m 8295Mi ds-cts-0 8773m 11978Mi ds-cts-1 8973m 11970Mi ds-cts-2 8698m 12032Mi ds-idrepo-0 4805m 3257Mi ds-idrepo-1 4773m 3111Mi ds-idrepo-2 4717m 3037Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6911m 903Mi 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 INFO [loop_until]: kubectl --namespace=xlou top node 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- status code --- 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- stderr --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 INFO [http_cmd]: http status code OK 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG --- http response --- 00:49:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG 00:49:44 DEBUG --- status code --- 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:49:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG { 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG --- http response --- 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG { 00:49:44 DEBUG { 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG { 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG { 00:49:44 DEBUG { 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG { 00:49:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:49:44 DEBUG { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG { 00:49:44 DEBUG { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG { 00:49:44 DEBUG { 00:49:44 DEBUG { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG "data": { 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG ----- output ----- 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "data": { 00:49:44 DEBUG "data": { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "data": { 00:49:44 DEBUG { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "data": { 00:49:44 DEBUG "data": { 00:49:44 DEBUG { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "result": [ 00:49:44 DEBUG { 00:49:44 DEBUG "data": { 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "data": { 00:49:44 DEBUG "data": { 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "data": { 00:49:44 DEBUG "data": { 00:49:44 DEBUG "data": { 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "result": [ 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "data": { 00:49:44 DEBUG { 00:49:44 DEBUG "status": "success", 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "result": [ 00:49:44 DEBUG "result": [ 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "result": [ 00:49:44 DEBUG "data": { 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG { 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "data": { 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "data": { 00:49:44 DEBUG "result": [ 00:49:44 DEBUG { 00:49:44 DEBUG { 00:49:44 DEBUG "result": [ 00:49:44 DEBUG "result": [ 00:49:44 DEBUG { 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "result": [ 00:49:44 DEBUG "result": [ 00:49:44 DEBUG "result": [ 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "result": [ 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG "result": [ 00:49:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:49:44 DEBUG "resultType": "vector", 00:49:44 DEBUG { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG { 00:49:44 DEBUG { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "result": [ 00:49:44 DEBUG { 00:49:44 DEBUG { 00:49:44 DEBUG { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG { 00:49:44 DEBUG "result": [ 00:49:44 DEBUG { 00:49:44 DEBUG }, 00:49:44 DEBUG "result": [ 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:49:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "pod": "ds-cts-0" 00:49:44 DEBUG { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:49:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:49:44 DEBUG "metric": { 00:49:44 DEBUG { 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "value": [ 00:49:44 DEBUG { 00:49:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:49:44 DEBUG }, 00:49:44 DEBUG }, 00:49:44 DEBUG "pod": "ds-idrepo-0" 00:49:44 DEBUG "instance": "10.142.0.107:9100" 00:49:44 DEBUG }, 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:49:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:49:44 DEBUG }, 00:49:44 DEBUG }, 00:49:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:49:44 DEBUG "metric": { 00:49:44 DEBUG "pod": "ds-cts-0" 00:49:44 DEBUG 00:49:44 DEBUG "metric": { 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG "value": [ 00:49:44 DEBUG }, 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG "pod": "ds-idrepo-0" 00:49:44 DEBUG }, 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG "value": [ 00:49:44 DEBUG }, 00:49:44 DEBUG "instance": "10.142.0.107:9100" 00:49:44 DEBUG }, 00:49:44 DEBUG 00:49:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "value": [ 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "value": [ 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG }, 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG [...] 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG }, 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:49:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "pod": "ds-cts-2" 00:49:44 DEBUG 00:49:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:49:44 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG "pod": "ds-cts-2" 00:49:44 DEBUG 00:49:44 DEBUG 00:49:44 DEBUG }, 00:49:44 DEBUG }, 00:49:44 DEBUG "pod": "ds-idrepo-2" 00:49:44 DEBUG 00:49:44 DEBUG "instance": "10.142.0.43:9100" 00:49:44 DEBUG "pod": "ds-idrepo-2" 00:49:44 DEBUG }, 00:49:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:49:44 DEBUG }, 00:49:44 DEBUG }, 00:49:44 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:49:44 DEBUG 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG }, 00:49:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:49:44 DEBUG "value": [ 00:49:44 DEBUG "value": [ 00:49:44 DEBUG }, 00:49:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:49:44 DEBUG }, 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG "value": [ 00:49:44 DEBUG }, 00:49:44 DEBUG "instance": "10.142.0.43:9100" 00:49:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:49:44 DEBUG "691515" 00:49:44 DEBUG "value": [ 00:49:44 DEBUG }, 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG }, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG }, 00:49:44 DEBUG }, 00:49:44 DEBUG ] 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG "1928.1" 00:49:44 DEBUG "1928.0666666666666" 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "0.000056431111111111105" 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "1043845120" 00:49:44 DEBUG "0" 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "value": [ 00:49:44 DEBUG "value": [ 00:49:44 DEBUG } 00:49:44 DEBUG "0" 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG ] 00:49:44 DEBUG ] 00:49:44 DEBUG "0" 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG "49872984.733333334" 00:49:44 DEBUG "0" 00:49:44 DEBUG ] 00:49:44 DEBUG "1927.9666666666667" 00:49:44 DEBUG ] 00:49:44 DEBUG ] 00:49:44 DEBUG "13.890515959872417" 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG 1713484184, 00:49:44 DEBUG ] 00:49:44 DEBUG ] 00:49:44 DEBUG "16.733" 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG "0" 00:49:44 DEBUG ] 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG "0.02966666666666675" 00:49:44 DEBUG "6.946489531058918" 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG ] 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG ] 00:49:44 DEBUG ] 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 DEBUG } 00:49:44 INFO [loop_until]: OK (rc = 0) 00:49:44 DEBUG --- stdout --- 00:49:44 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5925Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4795m 16% 6648Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8949m 29% 11590Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3215Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9125m 30% 14409Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5211Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10658m 35% 12161Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28119m 94% 3147Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10481m 35% 10486Mi 9% gke-xlou-cdm-ds-0f601751-j488 5065m 16% 6093Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8930m 29% 15935Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5390m 18% 6751Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8297m 27% 13893Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7139m 44% 4666Mi 7% 00:49:44 DEBUG --- stderr --- 00:49:44 DEBUG 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484244" 00:50:44 ERROR [http_cmd]: ERROR 00:50:44 ERROR --- status code --- 00:50:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:50:44 ERROR --- http response --- 00:50:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:50:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG { 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "data": { 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "result": [ 00:50:44 DEBUG { 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "instance": "10.142.0.107:9100" 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG "instance": "10.142.0.43:9100" 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG "0.06740000000000256" 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG { 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "data": { 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "result": [ 00:50:44 DEBUG { 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG "500" 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG { 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "data": { 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "result": [ 00:50:44 DEBUG { 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "instance": "10.142.0.107:9100" 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG "instance": "10.142.0.43:9100" 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG "288921517056" 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG { 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "data": { 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "result": [ 00:50:44 DEBUG { 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG "0" 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG { 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "data": { 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "result": [ 00:50:44 DEBUG { 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "pod": "ds-idrepo-0" 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG "pod": "ds-cts-2" 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG "0" 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484244" 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG { 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "data": { 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "result": [ 00:50:44 DEBUG { 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "instance": "10.142.0.107:9100" 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG "instance": "10.142.0.43:9100" 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG "49661010.599999994" 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG --- http response --- 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG { 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "data": { 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "result": [ 00:50:44 DEBUG { 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "pod": "ds-cts-0" 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG }, 00:50:44 DEBUG { 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG 00:50:44 DEBUG "data": { 00:50:44 DEBUG 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG 00:50:44 DEBUG "result": [ 00:50:44 DEBUG [...] 00:50:44 DEBUG { 00:50:44 DEBUG 00:50:44 DEBUG "metric": { 00:50:44 DEBUG 00:50:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:50:44 DEBUG 00:50:44 DEBUG }, 00:50:44 DEBUG "pod": "ds-cts-2" 00:50:44 DEBUG "value": [ 00:50:44 DEBUG }, 00:50:44 DEBUG 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG 00:50:44 DEBUG "0" 00:50:44 DEBUG [...] 00:50:44 DEBUG ] 00:50:44 DEBUG 00:50:44 DEBUG } 00:50:44 DEBUG 00:50:44 DEBUG ] 00:50:44 DEBUG 00:50:44 DEBUG } 00:50:44 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:50:44 DEBUG } 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG "77208.30224021644" 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG { 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "data": { 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "result": [ 00:50:44 DEBUG { 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "pod": "ds-idrepo-0" 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG "pod": "ds-idrepo-2" 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG "0" 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 INFO 00:50:44 INFO [loop_until]: kubectl --namespace=xlou top pods 00:50:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG { 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG "data": { 00:50:44 DEBUG { 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "result": [ 00:50:44 DEBUG "data": { 00:50:44 DEBUG { 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "result": [ 00:50:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:50:44 DEBUG { 00:50:44 DEBUG }, 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:50:44 DEBUG 00:50:44 DEBUG }, 00:50:44 DEBUG 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:50:44 DEBUG 00:50:44 DEBUG }, 00:50:44 DEBUG 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG }, 00:50:44 DEBUG "1911.2333333333333" 00:50:44 DEBUG "value": [ 00:50:44 DEBUG ] 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG } 00:50:44 DEBUG "11.617995416767766" 00:50:44 DEBUG ] 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484244" 00:50:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484244" 00:50:44 INFO [loop_until]: OK (rc = 0) 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG --- stdout --- 00:50:44 DEBUG --- status code --- 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG --- status code --- 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10676m 9367Mi am-6fd554f8f7-hvvkw 8830m 8968Mi am-6fd554f8f7-pmj4b 9361m 9042Mi ds-cts-0 8872m 11984Mi ds-cts-1 10957m 11985Mi ds-cts-2 9028m 12042Mi ds-idrepo-0 4890m 3257Mi ds-idrepo-1 4772m 3107Mi ds-idrepo-2 4583m 3038Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 14m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6774m 905Mi 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG --- stderr --- 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG --- http response --- 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:50:44 DEBUG --- http response --- 00:50:44 INFO 00:50:44 DEBUG --- http response --- 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 INFO [http_cmd]: http status code OK 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG ----- output ----- 00:50:44 INFO [loop_until]: kubectl --namespace=xlou top node 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG --- status code --- 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG { 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:44 DEBUG { 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:50:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:50:44 DEBUG { 00:50:44 DEBUG { 00:50:44 DEBUG { 00:50:44 DEBUG { 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG { 00:50:44 DEBUG { 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG --- http response --- 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "data": { 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:50:44 DEBUG { 00:50:44 DEBUG { 00:50:44 DEBUG "data": { 00:50:44 DEBUG { 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:50:44 DEBUG "data": { 00:50:44 DEBUG "data": { 00:50:44 DEBUG "data": { 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "data": { 00:50:44 DEBUG "data": { 00:50:44 DEBUG "data": { 00:50:44 DEBUG { 00:50:44 DEBUG ----- output ----- 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "result": [ 00:50:44 DEBUG { 00:50:44 DEBUG "data": { 00:50:44 DEBUG "data": { 00:50:44 DEBUG "result": [ 00:50:44 DEBUG "data": { 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG { 00:50:44 DEBUG "result": [ 00:50:44 DEBUG "result": [ 00:50:44 DEBUG "result": [ 00:50:44 DEBUG { 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG { 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "result": [ 00:50:44 DEBUG "result": [ 00:50:44 DEBUG "result": [ 00:50:44 DEBUG "data": { 00:50:44 DEBUG "status": "success", 00:50:44 DEBUG { 00:50:44 DEBUG { 00:50:44 DEBUG { 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "data": { 00:50:44 DEBUG "result": [ 00:50:44 DEBUG "result": [ 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "result": [ 00:50:44 DEBUG { 00:50:44 DEBUG { 00:50:44 DEBUG { 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "data": { 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "pod": "ds-cts-0" 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG { 00:50:44 DEBUG { 00:50:44 DEBUG "pod": "ds-cts-0" 00:50:44 DEBUG { 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "result": [ 00:50:44 DEBUG "resultType": "vector", 00:50:44 DEBUG "metric": { 00:50:44 DEBUG }, 00:50:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:50:44 DEBUG }, 00:50:44 DEBUG "result": [ 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "metric": { 00:50:44 DEBUG }, 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:50:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:50:44 DEBUG "pod": "ds-idrepo-0" 00:50:44 DEBUG { 00:50:44 DEBUG "result": [ 00:50:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:50:44 DEBUG "value": [ 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG { 00:50:44 DEBUG "pod": "ds-idrepo-0" 00:50:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:50:44 DEBUG }, 00:50:44 DEBUG }, 00:50:44 DEBUG }, 00:50:44 DEBUG "metric": { 00:50:44 DEBUG { 00:50:44 DEBUG }, 00:50:44 DEBUG 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG "metric": { 00:50:44 DEBUG }, 00:50:44 DEBUG }, 00:50:44 DEBUG 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "instance": "10.142.0.107:9100" 00:50:44 DEBUG "metric": { 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG }, 00:50:44 DEBUG "instance": "10.142.0.107:9100" 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG }, 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG "value": [ 00:50:44 DEBUG }, 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG [...] 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:50:44 DEBUG 00:50:44 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:50:44 DEBUG "pod": "ds-cts-2" 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG "pod": "ds-cts-2" 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG [...] 00:50:44 DEBUG 00:50:44 DEBUG }, 00:50:44 DEBUG [...] 00:50:44 DEBUG }, 00:50:44 DEBUG }, 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG }, 00:50:44 DEBUG 00:50:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:50:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:50:44 DEBUG "pod": "ds-idrepo-2" 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG "pod": "ds-idrepo-2" 00:50:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:50:44 DEBUG }, 00:50:44 DEBUG }, 00:50:44 DEBUG }, 00:50:44 DEBUG 00:50:44 DEBUG 00:50:44 DEBUG }, 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG 00:50:44 DEBUG "0" 00:50:44 DEBUG "value": [ 00:50:44 DEBUG }, 00:50:44 DEBUG }, 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG }, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "instance": "10.142.0.43:9100" 00:50:44 DEBUG 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "806417.5" 00:50:44 DEBUG 00:50:44 DEBUG ] 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "value": [ 00:50:44 DEBUG "50.733333333333334" 00:50:44 DEBUG "value": [ 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG }, 00:50:44 DEBUG "instance": "10.142.0.43:9100" 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG ] 00:50:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:50:44 DEBUG } 00:50:44 DEBUG "0.00006154888888888889" 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG ] 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG "18.2975" 00:50:44 DEBUG "1911.2666666666667" 00:50:44 DEBUG "0" 00:50:44 DEBUG "value": [ 00:50:44 DEBUG }, 00:50:44 DEBUG "1043832832" 00:50:44 DEBUG } 00:50:44 DEBUG }, 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG "0" 00:50:44 DEBUG } 00:50:44 DEBUG "6.963951207128672" 00:50:44 DEBUG ] 00:50:44 DEBUG ] 00:50:44 DEBUG ] 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG "value": [ 00:50:44 DEBUG ] 00:50:44 DEBUG ] 00:50:44 DEBUG "value": [ 00:50:44 DEBUG ] 00:50:44 DEBUG "0" 00:50:44 DEBUG ] 00:50:44 DEBUG ] 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG "0.04166666666666628" 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG "49665990.86666667" 00:50:44 DEBUG } 00:50:44 DEBUG 1713484244, 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG "1910.8333333333333" 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG ] 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 DEBUG } 00:50:44 INFO [loop_until]: OK (rc = 0) 00:50:44 DEBUG --- stdout --- 00:50:44 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5931Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4703m 15% 6638Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8940m 29% 11696Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3208Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8931m 29% 14406Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 168m 0% 5220Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10650m 35% 12165Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27760m 92% 3127Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10079m 33% 11254Mi 10% gke-xlou-cdm-ds-0f601751-j488 5056m 16% 6096Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 10230m 34% 15953Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5256m 17% 6734Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8656m 28% 13909Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 6872m 43% 4667Mi 7% 00:50:44 DEBUG --- stderr --- 00:50:44 DEBUG 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484304" 00:51:44 ERROR [http_cmd]: ERROR 00:51:44 ERROR --- status code --- 00:51:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:51:44 ERROR --- http response --- 00:51:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:51:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484304" 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG { 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "data": { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "result": [ 00:51:44 DEBUG { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "instance": "10.142.0.107:9100" 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "instance": "10.142.0.43:9100" 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG "0.09946666666666507" 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG { 00:51:44 DEBUG { 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "data": { 00:51:44 DEBUG "data": { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "result": [ 00:51:44 DEBUG "result": [ 00:51:44 DEBUG { 00:51:44 DEBUG { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:51:44 DEBUG "instance": "10.142.0.107:9100" 00:51:44 DEBUG }, 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:51:44 DEBUG "instance": "10.142.0.43:9100" 00:51:44 DEBUG }, 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG "166.66666666666666" 00:51:44 DEBUG "291240844288" 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG { 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "data": { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "result": [ 00:51:44 DEBUG { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "pod": "ds-cts-0" 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG "39121741.04755933" 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG { 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "data": { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "result": [ 00:51:44 DEBUG { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "pod": "ds-idrepo-0" 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "pod": "ds-cts-2" 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG "0" 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484304" 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG { 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "data": { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "result": [ 00:51:44 DEBUG { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "instance": "10.142.0.107:9100" 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "instance": "10.142.0.43:9100" 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG "49139741.13333333" 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG --- http response --- 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG { 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG "data": { 00:51:44 DEBUG { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "result": [ 00:51:44 DEBUG "data": { 00:51:44 DEBUG { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "result": [ 00:51:44 DEBUG "pod": "ds-cts-0" 00:51:44 DEBUG { 00:51:44 DEBUG }, 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "value": [ 00:51:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:51:44 DEBUG 00:51:44 DEBUG }, 00:51:44 DEBUG 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "pod": "ds-cts-2" 00:51:44 DEBUG 00:51:44 DEBUG }, 00:51:44 DEBUG 00:51:44 DEBUG "value": [ 00:51:44 DEBUG "pod": "ds-cts-0" 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG }, 00:51:44 DEBUG "1.2" 00:51:44 DEBUG "value": [ 00:51:44 DEBUG ] 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG } 00:51:44 DEBUG "192640.02112280935" 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG { 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "data": { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "result": [ 00:51:44 DEBUG { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "pod": "ds-idrepo-0" 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "pod": "ds-idrepo-2" 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG "0" 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG { 00:51:44 DEBUG { 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "data": { 00:51:44 DEBUG "data": { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "result": [ 00:51:44 DEBUG "result": [ 00:51:44 DEBUG { 00:51:44 DEBUG { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:51:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:51:44 DEBUG }, 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:51:44 DEBUG "pod": "ds-cts-0" 00:51:44 DEBUG }, 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG "1915.0666666666666" 00:51:44 DEBUG "8.983312386547281" 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO 00:51:44 INFO [loop_until]: kubectl --namespace=xlou top pods 00:51:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG { 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "data": { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "result": [ 00:51:44 DEBUG { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG "0" 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484304" 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG { 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "data": { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "result": [ 00:51:44 DEBUG { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG "6.9456296249867675" 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484304" 00:51:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484304" 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG --- status code --- 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG --- status code --- 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG --- status code --- 00:51:44 INFO [http_cmd]: http status code OK 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG --- status code --- 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:51:44 DEBUG { 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:44 DEBUG --- http response --- 00:51:44 DEBUG { 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG { 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:51:44 DEBUG { 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG { 00:51:44 DEBUG "data": { 00:51:44 DEBUG { 00:51:44 DEBUG { 00:51:44 DEBUG { 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG "data": { 00:51:44 DEBUG ----- output ----- 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG { 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG { 00:51:44 DEBUG "data": { 00:51:44 DEBUG { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG { 00:51:44 DEBUG "data": { 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "data": { 00:51:44 DEBUG "result": [ 00:51:44 DEBUG "data": { 00:51:44 DEBUG "data": { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "result": [ 00:51:44 DEBUG "status": "success", 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "data": { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "data": { 00:51:44 DEBUG "result": [ 00:51:44 DEBUG "data": { 00:51:44 DEBUG "result": [ 00:51:44 DEBUG "data": { 00:51:44 DEBUG { 00:51:44 DEBUG "data": { 00:51:44 DEBUG "result": [ 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "result": [ 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "result": [ 00:51:44 DEBUG { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "result": [ 00:51:44 DEBUG { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG { 00:51:44 DEBUG "result": [ 00:51:44 DEBUG { 00:51:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:51:44 DEBUG { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "resultType": "vector", 00:51:44 DEBUG { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "result": [ 00:51:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:51:44 DEBUG "result": [ 00:51:44 DEBUG "metric": { 00:51:44 DEBUG { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "pod": "ds-cts-0" 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:51:44 DEBUG "result": [ 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "pod": "ds-cts-0" 00:51:44 DEBUG { 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:51:44 DEBUG "metric": { 00:51:44 DEBUG }, 00:51:44 DEBUG }, 00:51:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:51:44 DEBUG }, 00:51:44 DEBUG { 00:51:44 DEBUG "pod": "ds-idrepo-0" 00:51:44 DEBUG }, 00:51:44 DEBUG }, 00:51:44 DEBUG { 00:51:44 DEBUG 00:51:44 DEBUG }, 00:51:44 DEBUG "instance": "10.142.0.107:9100" 00:51:44 DEBUG "value": [ 00:51:44 DEBUG "value": [ 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "metric": { 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG "value": [ 00:51:44 DEBUG }, 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:51:44 DEBUG "pod": "ds-idrepo-0" 00:51:44 DEBUG "metric": { 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG }, 00:51:44 DEBUG }, 00:51:44 DEBUG "instance": "10.142.0.107:9100" 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "value": [ 00:51:44 DEBUG "value": [ 00:51:44 DEBUG }, 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "pod": "ds-cts-0" 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG }, 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG [...] 00:51:44 DEBUG 00:51:44 DEBUG "pod": "ds-idrepo-2" 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "instance": "10.142.0.43:9100" 00:51:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG [...] 00:51:44 DEBUG }, 00:51:44 INFO [loop_until]: OK (rc = 0) 00:51:44 DEBUG 00:51:44 DEBUG "pod": "ds-cts-2" 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG 00:51:44 DEBUG "pod": "ds-cts-2" 00:51:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:51:44 DEBUG }, 00:51:44 DEBUG }, 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG "value": [ 00:51:44 DEBUG --- stdout --- 00:51:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:51:44 DEBUG }, 00:51:44 DEBUG "455982.3096471831" 00:51:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:51:44 DEBUG }, 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10849m 9389Mi am-6fd554f8f7-hvvkw 8781m 8893Mi am-6fd554f8f7-pmj4b 9371m 9049Mi ds-cts-0 8977m 11993Mi ds-cts-1 9890m 11994Mi ds-cts-2 10013m 12056Mi ds-idrepo-0 4764m 3258Mi ds-idrepo-1 4724m 3110Mi ds-idrepo-2 4633m 3037Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6849m 906Mi 00:51:44 DEBUG ] 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:51:44 DEBUG "pod": "ds-idrepo-2" 00:51:44 DEBUG 00:51:44 DEBUG "0" 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG --- stderr --- 00:51:44 DEBUG } 00:51:44 DEBUG "value": [ 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG "49145104.06666666" 00:51:44 DEBUG "1046335488" 00:51:44 DEBUG }, 00:51:44 DEBUG }, 00:51:44 DEBUG "instance": "10.142.0.43:9100" 00:51:44 DEBUG ] 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG "10.133333333333333" 00:51:44 DEBUG ] 00:51:44 DEBUG 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG "0" 00:51:44 DEBUG "1914.9666666666667" 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 INFO 00:51:44 DEBUG "value": [ 00:51:44 DEBUG "value": [ 00:51:44 DEBUG }, 00:51:44 DEBUG "value": [ 00:51:44 DEBUG "921338.5" 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG "1914.9666666666667" 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 INFO [loop_until]: kubectl --namespace=xlou top node 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG 1713484304, 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:51:44 DEBUG "19.4705" 00:51:44 DEBUG "0" 00:51:44 DEBUG "0.06033333333333332" 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 DEBUG ] 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:44 DEBUG } 00:51:45 INFO [loop_until]: OK (rc = 0) 00:51:45 DEBUG --- stdout --- 00:51:45 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5935Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4691m 15% 6661Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8752m 29% 11635Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3216Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8375m 28% 14418Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5203Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10698m 35% 12215Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27857m 93% 3132Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10033m 33% 11268Mi 10% gke-xlou-cdm-ds-0f601751-j488 5038m 16% 6089Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 10251m 34% 15962Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5300m 17% 6737Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9892m 33% 13924Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7089m 44% 4664Mi 7% 00:51:45 DEBUG --- stderr --- 00:51:45 DEBUG 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484364" 00:52:44 ERROR [http_cmd]: ERROR 00:52:44 ERROR --- status code --- 00:52:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:52:44 ERROR --- http response --- 00:52:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:52:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484364" 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484364" 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484364" 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484364" 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484364" 00:52:44 INFO [http_cmd]: http status code OK 00:52:44 INFO [http_cmd]: http status code OK 00:52:44 DEBUG --- status code --- 00:52:44 DEBUG --- status code --- 00:52:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:44 DEBUG --- http response --- 00:52:44 DEBUG --- http response --- 00:52:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:44 INFO [http_cmd]: http status code OK 00:52:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:52:44 DEBUG ----- output ----- 00:52:44 DEBUG --- status code --- 00:52:44 DEBUG { 00:52:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:44 DEBUG "status": "success", 00:52:44 DEBUG --- http response --- 00:52:44 DEBUG "data": { 00:52:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:44 DEBUG "resultType": "vector", 00:52:44 DEBUG ----- output ----- 00:52:44 DEBUG "result": [ 00:52:44 DEBUG { 00:52:44 DEBUG { 00:52:44 DEBUG "status": "success", 00:52:44 DEBUG "metric": { 00:52:44 DEBUG "data": { 00:52:44 DEBUG "instance": "10.142.0.107:9100" 00:52:44 DEBUG "resultType": "vector", 00:52:44 DEBUG }, 00:52:44 DEBUG "result": [ 00:52:44 DEBUG { 00:52:44 DEBUG "metric": { 00:52:44 DEBUG "value": [ 00:52:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:52:44 DEBUG 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG [...] 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:52:44 DEBUG 00:52:44 DEBUG }, 00:52:44 DEBUG [...] 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 00:52:44 DEBUG 1713484364, 00:52:44 DEBUG 00:52:44 DEBUG "0" 00:52:44 DEBUG 00:52:44 DEBUG ] 00:52:44 DEBUG "instance": "10.142.0.43:9100" 00:52:44 DEBUG } 00:52:44 DEBUG }, 00:52:44 DEBUG ] 00:52:44 INFO [http_cmd]: http status code OK 00:52:44 DEBUG "value": [ 00:52:44 DEBUG } 00:52:44 DEBUG } 00:52:44 DEBUG 1713484364, 00:52:44 DEBUG --- status code --- 00:52:44 DEBUG "0.26766666666666383" 00:52:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:44 DEBUG ] 00:52:44 DEBUG --- http response --- 00:52:44 DEBUG } 00:52:44 INFO [http_cmd]: http status code OK 00:52:44 DEBUG ] 00:52:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:44 DEBUG --- status code --- 00:52:44 DEBUG } 00:52:44 DEBUG ----- output ----- 00:52:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:44 DEBUG } 00:52:44 DEBUG { 00:52:44 DEBUG --- http response --- 00:52:44 DEBUG "status": "success", 00:52:44 DEBUG "data": { 00:52:44 DEBUG "resultType": "vector", 00:52:44 DEBUG "result": [ 00:52:44 DEBUG { 00:52:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:52:44 DEBUG "metric": { 00:52:44 DEBUG ----- output ----- 00:52:44 DEBUG "instance": "10.142.0.107:9100" 00:52:44 DEBUG { 00:52:44 DEBUG }, 00:52:44 DEBUG "status": "success", 00:52:44 DEBUG "value": [ 00:52:44 DEBUG "data": { 00:52:44 DEBUG 00:52:44 DEBUG "resultType": "vector", 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG [...] 00:52:44 DEBUG 00:52:44 DEBUG "result": [ 00:52:44 DEBUG 00:52:44 DEBUG { 00:52:44 DEBUG 00:52:44 DEBUG "metric": { 00:52:44 DEBUG "instance": "10.142.0.43:9100" 00:52:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:52:44 DEBUG }, 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 1713484364, 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG [...] 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 1713484364, 00:52:44 DEBUG "695.1209164191769" 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG } 00:52:44 DEBUG "293576686592" 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG } 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484364" 00:52:44 INFO [http_cmd]: http status code OK 00:52:44 DEBUG --- status code --- 00:52:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:44 DEBUG --- http response --- 00:52:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:52:44 DEBUG ----- output ----- 00:52:44 DEBUG { 00:52:44 DEBUG "status": "success", 00:52:44 DEBUG "data": { 00:52:44 DEBUG "resultType": "vector", 00:52:44 DEBUG "result": [ 00:52:44 DEBUG { 00:52:44 DEBUG "metric": { 00:52:44 DEBUG "pod": "ds-idrepo-0" 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG [...] 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG "pod": "ds-cts-2" 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 1713484364, 00:52:44 DEBUG "0" 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG } 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484364" 00:52:44 INFO [http_cmd]: http status code OK 00:52:44 DEBUG --- status code --- 00:52:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:44 DEBUG --- http response --- 00:52:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:44 DEBUG ----- output ----- 00:52:44 DEBUG { 00:52:44 DEBUG "status": "success", 00:52:44 DEBUG "data": { 00:52:44 DEBUG "resultType": "vector", 00:52:44 DEBUG "result": [ 00:52:44 DEBUG { 00:52:44 DEBUG "metric": { 00:52:44 DEBUG "instance": "10.142.0.107:9100" 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG [...] 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG "instance": "10.142.0.43:9100" 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 1713484364, 00:52:44 DEBUG "48020163.06666666" 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG } 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484364" 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484364" 00:52:44 INFO [http_cmd]: http status code OK 00:52:44 DEBUG --- status code --- 00:52:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:44 DEBUG --- http response --- 00:52:44 INFO [http_cmd]: http status code OK 00:52:44 DEBUG --- status code --- 00:52:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:44 DEBUG --- http response --- 00:52:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:52:44 DEBUG ----- output ----- 00:52:44 DEBUG { 00:52:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:52:44 DEBUG "status": "success", 00:52:44 DEBUG ----- output ----- 00:52:44 DEBUG "data": { 00:52:44 DEBUG { 00:52:44 DEBUG "resultType": "vector", 00:52:44 DEBUG "status": "success", 00:52:44 DEBUG "result": [ 00:52:44 DEBUG "data": { 00:52:44 DEBUG { 00:52:44 DEBUG "resultType": "vector", 00:52:44 DEBUG "metric": { 00:52:44 DEBUG "result": [ 00:52:44 DEBUG "pod": "ds-cts-0" 00:52:44 DEBUG { 00:52:44 DEBUG }, 00:52:44 DEBUG "metric": { 00:52:44 DEBUG "value": [ 00:52:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:52:44 DEBUG 00:52:44 DEBUG }, 00:52:44 DEBUG 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG [...] 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG [...] 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG "pod": "ds-cts-2" 00:52:44 DEBUG 00:52:44 DEBUG }, 00:52:44 DEBUG 00:52:44 DEBUG "value": [ 00:52:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:52:44 DEBUG 1713484364, 00:52:44 DEBUG }, 00:52:44 DEBUG "1.5" 00:52:44 DEBUG "value": [ 00:52:44 DEBUG ] 00:52:44 DEBUG 1713484364, 00:52:44 DEBUG } 00:52:44 DEBUG "11990.8358082308" 00:52:44 DEBUG ] 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG } 00:52:44 DEBUG } 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG } 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484364" 00:52:44 INFO [http_cmd]: http status code OK 00:52:44 DEBUG --- status code --- 00:52:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:44 DEBUG --- http response --- 00:52:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:44 DEBUG ----- output ----- 00:52:44 DEBUG { 00:52:44 DEBUG "status": "success", 00:52:44 DEBUG "data": { 00:52:44 DEBUG "resultType": "vector", 00:52:44 DEBUG "result": [ 00:52:44 DEBUG { 00:52:44 DEBUG "metric": { 00:52:44 DEBUG "pod": "ds-idrepo-0" 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG [...] 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG "pod": "ds-idrepo-2" 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 1713484364, 00:52:44 DEBUG "0" 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG } 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484364" 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484364" 00:52:44 INFO [http_cmd]: http status code OK 00:52:44 DEBUG --- status code --- 00:52:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:44 DEBUG --- http response --- 00:52:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:52:44 DEBUG ----- output ----- 00:52:44 DEBUG { 00:52:44 DEBUG "status": "success", 00:52:44 DEBUG "data": { 00:52:44 DEBUG "resultType": "vector", 00:52:44 DEBUG "result": [ 00:52:44 DEBUG { 00:52:44 DEBUG "metric": { 00:52:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG [...] 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 1713484364, 00:52:44 DEBUG "9.177281417055566" 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG } 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484364" 00:52:44 INFO [http_cmd]: http status code OK 00:52:44 DEBUG --- status code --- 00:52:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:44 DEBUG --- http response --- 00:52:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:44 DEBUG ----- output ----- 00:52:44 DEBUG { 00:52:44 DEBUG "status": "success", 00:52:44 DEBUG "data": { 00:52:44 DEBUG "resultType": "vector", 00:52:44 DEBUG "result": [ 00:52:44 DEBUG { 00:52:44 DEBUG "metric": { 00:52:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 00:52:44 INFO [http_cmd]: http status code OK 00:52:44 DEBUG 00:52:44 DEBUG --- status code --- 00:52:44 DEBUG 00:52:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:44 DEBUG [...] 00:52:44 DEBUG --- http response --- 00:52:44 DEBUG 00:52:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:44 DEBUG 00:52:44 DEBUG ----- output ----- 00:52:44 DEBUG { 00:52:44 DEBUG 00:52:44 DEBUG "status": "success", 00:52:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:52:44 DEBUG "data": { 00:52:44 DEBUG }, 00:52:44 DEBUG "resultType": "vector", 00:52:44 DEBUG "value": [ 00:52:44 DEBUG "result": [ 00:52:44 DEBUG 1713484364, 00:52:44 DEBUG { 00:52:44 DEBUG "1921.5666666666666" 00:52:44 DEBUG "metric": { 00:52:44 DEBUG ] 00:52:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:52:44 DEBUG } 00:52:44 DEBUG }, 00:52:44 DEBUG ] 00:52:44 DEBUG "value": [ 00:52:44 DEBUG } 00:52:44 DEBUG 00:52:44 DEBUG } 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG [...] 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 1713484364, 00:52:44 DEBUG "7.027693585502122" 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG } 00:52:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484364" 00:52:44 INFO [http_cmd]: http status code OK 00:52:44 DEBUG --- status code --- 00:52:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:44 DEBUG --- http response --- 00:52:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:44 DEBUG ----- output ----- 00:52:44 DEBUG { 00:52:44 DEBUG "status": "success", 00:52:44 DEBUG "data": { 00:52:44 DEBUG "resultType": "vector", 00:52:44 DEBUG "result": [ 00:52:44 DEBUG { 00:52:44 DEBUG "metric": { 00:52:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG [...] 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG 00:52:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:52:44 DEBUG }, 00:52:44 DEBUG "value": [ 00:52:44 DEBUG 1713484364, 00:52:44 DEBUG "0" 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG ] 00:52:44 DEBUG } 00:52:44 DEBUG } 00:52:45 INFO 00:52:45 INFO [loop_until]: kubectl --namespace=xlou top pods 00:52:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:52:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484365" 00:52:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484365" 00:52:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484365" 00:52:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484365" 00:52:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484365" 00:52:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484365" 00:52:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484365" 00:52:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484365" 00:52:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484365" 00:52:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484365" 00:52:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484365" 00:52:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484365" 00:52:45 INFO [http_cmd]: http status code OK 00:52:45 INFO [http_cmd]: http status code OK 00:52:45 INFO [http_cmd]: http status code OK 00:52:45 INFO [http_cmd]: http status code OK 00:52:45 INFO [http_cmd]: http status code OK 00:52:45 INFO [http_cmd]: http status code OK 00:52:45 DEBUG --- status code --- 00:52:45 INFO [http_cmd]: http status code OK 00:52:45 DEBUG --- status code --- 00:52:45 INFO [http_cmd]: http status code OK 00:52:45 DEBUG --- status code --- 00:52:45 INFO [http_cmd]: http status code OK 00:52:45 DEBUG --- status code --- 00:52:45 DEBUG --- status code --- 00:52:45 DEBUG --- status code --- 00:52:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:45 DEBUG --- status code --- 00:52:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:45 DEBUG --- status code --- 00:52:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:45 DEBUG --- status code --- 00:52:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:45 INFO [http_cmd]: http status code OK 00:52:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:45 DEBUG --- http response --- 00:52:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:45 DEBUG --- http response --- 00:52:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:45 DEBUG --- http response --- 00:52:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:45 DEBUG --- http response --- 00:52:45 INFO [http_cmd]: http status code OK 00:52:45 DEBUG --- status code --- 00:52:45 DEBUG --- http response --- 00:52:45 DEBUG --- http response --- 00:52:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:52:45 DEBUG --- http response --- 00:52:45 DEBUG --- http response --- 00:52:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:45 DEBUG --- http response --- 00:52:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:52:45 DEBUG --- status code --- 00:52:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:45 INFO [http_cmd]: http status code OK 00:52:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:52:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:45 DEBUG ----- output ----- 00:52:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:45 DEBUG ----- output ----- 00:52:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:45 DEBUG ----- output ----- 00:52:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:45 DEBUG ----- output ----- 00:52:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:45 DEBUG --- http response --- 00:52:45 DEBUG --- status code --- 00:52:45 DEBUG ----- output ----- 00:52:45 DEBUG ----- output ----- 00:52:45 DEBUG { 00:52:45 DEBUG ----- output ----- 00:52:45 DEBUG { 00:52:45 DEBUG ----- output ----- 00:52:45 DEBUG { 00:52:45 DEBUG ----- output ----- 00:52:45 DEBUG { 00:52:45 DEBUG --- http response --- 00:52:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:52:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:52:45 DEBUG { 00:52:45 DEBUG { 00:52:45 DEBUG "status": "success", 00:52:45 DEBUG { 00:52:45 DEBUG "status": "success", 00:52:45 DEBUG { 00:52:45 DEBUG "status": "success", 00:52:45 DEBUG { 00:52:45 DEBUG "status": "success", 00:52:45 DEBUG ----- output ----- 00:52:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:45 DEBUG --- http response --- 00:52:45 DEBUG "status": "success", 00:52:45 DEBUG "status": "success", 00:52:45 DEBUG "data": { 00:52:45 DEBUG "status": "success", 00:52:45 DEBUG "data": { 00:52:45 DEBUG "status": "success", 00:52:45 DEBUG "data": { 00:52:45 DEBUG "status": "success", 00:52:45 DEBUG "data": { 00:52:45 DEBUG { 00:52:45 DEBUG ----- output ----- 00:52:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:52:45 DEBUG ----- output ----- 00:52:45 DEBUG "data": { 00:52:45 DEBUG "resultType": "vector", 00:52:45 DEBUG "data": { 00:52:45 DEBUG "resultType": "vector", 00:52:45 DEBUG "data": { 00:52:45 DEBUG "resultType": "vector", 00:52:45 DEBUG "data": { 00:52:45 DEBUG "resultType": "vector", 00:52:45 DEBUG "status": "success", 00:52:45 DEBUG { 00:52:45 DEBUG { 00:52:45 DEBUG "data": { 00:52:45 DEBUG "resultType": "vector", 00:52:45 DEBUG "result": [ 00:52:45 DEBUG "resultType": "vector", 00:52:45 DEBUG "result": [ 00:52:45 DEBUG "resultType": "vector", 00:52:45 DEBUG "result": [ 00:52:45 DEBUG "resultType": "vector", 00:52:45 DEBUG "result": [ 00:52:45 DEBUG "data": { 00:52:45 DEBUG "status": "success", 00:52:45 DEBUG "status": "success", 00:52:45 DEBUG "resultType": "vector", 00:52:45 DEBUG "result": [ 00:52:45 DEBUG { 00:52:45 DEBUG "result": [ 00:52:45 DEBUG { 00:52:45 DEBUG "result": [ 00:52:45 DEBUG { 00:52:45 DEBUG "result": [ 00:52:45 DEBUG { 00:52:45 DEBUG "resultType": "vector", 00:52:45 DEBUG "data": { 00:52:45 DEBUG "data": { 00:52:45 DEBUG "result": [ 00:52:45 DEBUG { 00:52:45 DEBUG "metric": { 00:52:45 DEBUG { 00:52:45 DEBUG "metric": { 00:52:45 DEBUG { 00:52:45 DEBUG "metric": { 00:52:45 DEBUG { 00:52:45 DEBUG "metric": { 00:52:45 DEBUG "result": [ 00:52:45 DEBUG "resultType": "vector", 00:52:45 DEBUG "resultType": "vector", 00:52:45 DEBUG "result": [ 00:52:45 DEBUG "metric": { 00:52:45 DEBUG "pod": "ds-idrepo-0" 00:52:45 DEBUG "metric": { 00:52:45 DEBUG "instance": "10.142.0.107:9100" 00:52:45 DEBUG "metric": { 00:52:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:52:45 DEBUG "metric": { 00:52:45 DEBUG "pod": "ds-cts-0" 00:52:45 DEBUG { 00:52:45 DEBUG "result": [ 00:52:45 DEBUG { 00:52:45 DEBUG { 00:52:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:52:45 DEBUG }, 00:52:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:52:45 DEBUG }, 00:52:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:52:45 DEBUG }, 00:52:45 DEBUG "pod": "ds-idrepo-0" 00:52:45 DEBUG }, 00:52:45 DEBUG "metric": { 00:52:45 DEBUG { 00:52:45 DEBUG "metric": { 00:52:45 DEBUG "metric": { 00:52:45 DEBUG }, 00:52:45 DEBUG "value": [ 00:52:45 DEBUG }, 00:52:45 DEBUG "value": [ 00:52:45 DEBUG }, 00:52:45 DEBUG "value": [ 00:52:45 DEBUG }, 00:52:45 DEBUG "value": [ 00:52:45 DEBUG "instance": "10.142.0.107:9100" 00:52:45 DEBUG "metric": { 00:52:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:52:45 DEBUG "pod": "ds-cts-0" 00:52:45 DEBUG "value": [ 00:52:45 DEBUG 00:52:45 DEBUG "value": [ 00:52:45 DEBUG 00:52:45 DEBUG "value": [ 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG "value": [ 00:52:45 DEBUG }, 00:52:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:52:45 DEBUG }, 00:52:45 DEBUG }, 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG "value": [ 00:52:45 DEBUG }, 00:52:45 DEBUG "value": [ 00:52:45 DEBUG "value": [ 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG "value": [ 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG [...] 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG [...] 00:52:45 DEBUG 00:52:45 DEBUG [...] 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG [...] 00:52:45 DEBUG 00:52:45 DEBUG [...] 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG [...] 00:52:45 DEBUG 00:52:45 DEBUG [...] 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG [...] 00:52:45 DEBUG 00:52:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG [...] 00:52:45 DEBUG 00:52:45 DEBUG [...] 00:52:45 DEBUG [...] 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG }, 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG [...] 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG "value": [ 00:52:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:52:45 DEBUG 00:52:45 DEBUG "pod": "ds-idrepo-2" 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG "pod": "ds-idrepo-2" 00:52:45 DEBUG 00:52:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:52:45 DEBUG 1713484365, 00:52:45 INFO [loop_until]: OK (rc = 0) 00:52:45 DEBUG }, 00:52:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:52:45 DEBUG }, 00:52:45 DEBUG "pod": "ds-cts-2" 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG 00:52:45 DEBUG }, 00:52:45 DEBUG "instance": "10.142.0.43:9100" 00:52:45 DEBUG }, 00:52:45 DEBUG "0" 00:52:45 DEBUG --- stdout --- 00:52:45 DEBUG "value": [ 00:52:45 DEBUG }, 00:52:45 DEBUG "value": [ 00:52:45 DEBUG 1713484365, 00:52:45 DEBUG "instance": "10.142.0.43:9100" 00:52:45 DEBUG 00:52:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:52:45 DEBUG "pod": "ds-cts-2" 00:52:45 DEBUG "value": [ 00:52:45 DEBUG }, 00:52:45 DEBUG "value": [ 00:52:45 DEBUG ] 00:52:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10626m 9390Mi am-6fd554f8f7-hvvkw 8781m 8939Mi am-6fd554f8f7-pmj4b 9245m 9051Mi ds-cts-0 9525m 12009Mi ds-cts-1 9373m 12005Mi ds-cts-2 9305m 12056Mi ds-idrepo-0 4865m 3257Mi ds-idrepo-1 4862m 3109Mi ds-idrepo-2 4693m 3037Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6947m 909Mi 00:52:45 DEBUG 1713484365, 00:52:45 DEBUG "value": [ 00:52:45 DEBUG "0" 00:52:45 DEBUG }, 00:52:45 DEBUG }, 00:52:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:52:45 DEBUG }, 00:52:45 DEBUG }, 00:52:45 DEBUG 1713484365, 00:52:45 DEBUG "value": [ 00:52:45 DEBUG 1713484365, 00:52:45 DEBUG } 00:52:45 DEBUG --- stderr --- 00:52:45 DEBUG "1036081.5" 00:52:45 DEBUG 1713484365, 00:52:45 DEBUG ] 00:52:45 DEBUG "value": [ 00:52:45 DEBUG "value": [ 00:52:45 DEBUG }, 00:52:45 DEBUG "value": [ 00:52:45 DEBUG "value": [ 00:52:45 DEBUG "0" 00:52:45 DEBUG 1713484365, 00:52:45 DEBUG "1921.4666666666667" 00:52:45 DEBUG ] 00:52:45 DEBUG 00:52:45 DEBUG ] 00:52:45 DEBUG "1046777856" 00:52:45 DEBUG } 00:52:45 DEBUG 1713484365, 00:52:45 DEBUG 1713484365, 00:52:45 DEBUG "value": [ 00:52:45 DEBUG 1713484365, 00:52:45 DEBUG 1713484365, 00:52:45 DEBUG ] 00:52:45 DEBUG "48023768.5" 00:52:45 DEBUG ] 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG ] 00:52:45 DEBUG ] 00:52:45 DEBUG "51.699999999999996" 00:52:45 DEBUG "0.14366666666666647" 00:52:45 DEBUG 1713484365, 00:52:45 DEBUG "20.076" 00:52:45 DEBUG "0.00006666666666666672" 00:52:45 DEBUG } 00:52:45 DEBUG ] 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG ] 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG ] 00:52:45 DEBUG ] 00:52:45 DEBUG "1921.9333333333334" 00:52:45 DEBUG ] 00:52:45 DEBUG ] 00:52:45 DEBUG ] 00:52:45 DEBUG } 00:52:45 DEBUG ] 00:52:45 DEBUG } 00:52:45 DEBUG ] 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG ] 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG ] 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG ] 00:52:45 DEBUG ] 00:52:45 DEBUG } 00:52:45 DEBUG ] 00:52:45 DEBUG ] 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG ] 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 DEBUG } 00:52:45 INFO 00:52:45 INFO [loop_until]: kubectl --namespace=xlou top node 00:52:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:52:45 INFO [loop_until]: OK (rc = 0) 00:52:45 DEBUG --- stdout --- 00:52:45 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5925Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4622m 15% 6646Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8193m 27% 11659Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3223Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8980m 30% 14448Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 141m 0% 5199Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10660m 35% 12195Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27796m 93% 3120Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9884m 33% 11258Mi 10% gke-xlou-cdm-ds-0f601751-j488 5005m 16% 6085Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9523m 31% 15980Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5326m 17% 6750Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9867m 33% 13927Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7131m 44% 4663Mi 7% 00:52:45 DEBUG --- stderr --- 00:52:45 DEBUG 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484424" 00:53:44 ERROR [http_cmd]: ERROR 00:53:44 ERROR --- status code --- 00:53:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:53:44 ERROR --- http response --- 00:53:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:53:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484424" 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484424" 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484424" 00:53:44 INFO [http_cmd]: http status code OK 00:53:44 DEBUG --- status code --- 00:53:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:44 DEBUG --- http response --- 00:53:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:44 DEBUG ----- output ----- 00:53:44 DEBUG { 00:53:44 DEBUG "status": "success", 00:53:44 DEBUG "data": { 00:53:44 DEBUG "resultType": "vector", 00:53:44 DEBUG "result": [ 00:53:44 DEBUG { 00:53:44 DEBUG "metric": { 00:53:44 DEBUG "instance": "10.142.0.107:9100" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG [...] 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG "instance": "10.142.0.43:9100" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 1713484424, 00:53:44 DEBUG "0.6279666666666694" 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG } 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484424" 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484424" 00:53:44 INFO [http_cmd]: http status code OK 00:53:44 DEBUG --- status code --- 00:53:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:44 DEBUG --- http response --- 00:53:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:53:44 INFO [http_cmd]: http status code OK 00:53:44 INFO [http_cmd]: http status code OK 00:53:44 DEBUG --- status code --- 00:53:44 DEBUG --- status code --- 00:53:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:44 DEBUG --- http response --- 00:53:44 DEBUG --- http response --- 00:53:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:44 DEBUG ----- output ----- 00:53:44 DEBUG { 00:53:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:44 DEBUG "status": "success", 00:53:44 DEBUG ----- output ----- 00:53:44 DEBUG "data": { 00:53:44 DEBUG { 00:53:44 DEBUG "resultType": "vector", 00:53:44 DEBUG "status": "success", 00:53:44 DEBUG "result": [ 00:53:44 DEBUG "data": { 00:53:44 DEBUG "resultType": "vector", 00:53:44 DEBUG { 00:53:44 DEBUG "result": [ 00:53:44 DEBUG "metric": { 00:53:44 DEBUG { 00:53:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:53:44 DEBUG "metric": { 00:53:44 DEBUG }, 00:53:44 DEBUG "instance": "10.142.0.107:9100" 00:53:44 DEBUG "value": [ 00:53:44 DEBUG }, 00:53:44 DEBUG 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 00:53:44 INFO [http_cmd]: http status code OK 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG --- status code --- 00:53:44 DEBUG 00:53:44 DEBUG [...] 00:53:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG --- http response --- 00:53:44 DEBUG [...] 00:53:44 DEBUG 00:53:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG ----- output ----- 00:53:44 DEBUG 00:53:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:53:44 DEBUG { 00:53:44 DEBUG 00:53:44 DEBUG }, 00:53:44 DEBUG "status": "success", 00:53:44 DEBUG "instance": "10.142.0.43:9100" 00:53:44 DEBUG "value": [ 00:53:44 DEBUG "data": { 00:53:44 DEBUG }, 00:53:44 DEBUG 1713484424, 00:53:44 DEBUG "resultType": "vector", 00:53:44 DEBUG "value": [ 00:53:44 DEBUG "0" 00:53:44 DEBUG "result": [ 00:53:44 DEBUG 1713484424, 00:53:44 DEBUG ] 00:53:44 DEBUG { 00:53:44 DEBUG "295894653952" 00:53:44 DEBUG } 00:53:44 DEBUG "metric": { 00:53:44 DEBUG ] 00:53:44 DEBUG ] 00:53:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:53:44 DEBUG } 00:53:44 DEBUG } 00:53:44 DEBUG }, 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG "value": [ 00:53:44 DEBUG } 00:53:44 DEBUG 00:53:44 DEBUG } 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG [...] 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 1713484424, 00:53:44 DEBUG "0" 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG } 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484424" 00:53:44 INFO [http_cmd]: http status code OK 00:53:44 DEBUG --- status code --- 00:53:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:44 DEBUG --- http response --- 00:53:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:53:44 DEBUG ----- output ----- 00:53:44 DEBUG { 00:53:44 DEBUG "status": "success", 00:53:44 DEBUG "data": { 00:53:44 DEBUG "resultType": "vector", 00:53:44 DEBUG "result": [ 00:53:44 DEBUG { 00:53:44 DEBUG "metric": { 00:53:44 DEBUG "pod": "ds-idrepo-0" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG [...] 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG "pod": "ds-cts-2" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 1713484424, 00:53:44 DEBUG "0" 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG } 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484424" 00:53:44 INFO [http_cmd]: http status code OK 00:53:44 DEBUG --- status code --- 00:53:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:44 DEBUG --- http response --- 00:53:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:44 DEBUG ----- output ----- 00:53:44 DEBUG { 00:53:44 DEBUG "status": "success", 00:53:44 DEBUG "data": { 00:53:44 DEBUG "resultType": "vector", 00:53:44 DEBUG "result": [ 00:53:44 DEBUG { 00:53:44 DEBUG "metric": { 00:53:44 DEBUG "instance": "10.142.0.107:9100" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG [...] 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG "instance": "10.142.0.43:9100" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 1713484424, 00:53:44 DEBUG "47051046.266666666" 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG } 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484424" 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484424" 00:53:44 INFO [http_cmd]: http status code OK 00:53:44 INFO [http_cmd]: http status code OK 00:53:44 DEBUG --- status code --- 00:53:44 DEBUG --- status code --- 00:53:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:44 DEBUG --- http response --- 00:53:44 DEBUG --- http response --- 00:53:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:53:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:53:44 DEBUG ----- output ----- 00:53:44 DEBUG ----- output ----- 00:53:44 DEBUG { 00:53:44 DEBUG { 00:53:44 DEBUG "status": "success", 00:53:44 DEBUG "status": "success", 00:53:44 DEBUG "data": { 00:53:44 DEBUG "data": { 00:53:44 DEBUG "resultType": "vector", 00:53:44 DEBUG "resultType": "vector", 00:53:44 DEBUG "result": [ 00:53:44 DEBUG "result": [ 00:53:44 DEBUG { 00:53:44 DEBUG { 00:53:44 DEBUG "metric": { 00:53:44 DEBUG "metric": { 00:53:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG "pod": "ds-cts-0" 00:53:44 DEBUG }, 00:53:44 DEBUG 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG [...] 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG [...] 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:53:44 DEBUG 00:53:44 DEBUG }, 00:53:44 DEBUG "pod": "ds-cts-2" 00:53:44 DEBUG "value": [ 00:53:44 DEBUG }, 00:53:44 DEBUG 1713484424, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG "0" 00:53:44 DEBUG 1713484424, 00:53:44 DEBUG ] 00:53:44 DEBUG "1.3666666666666667" 00:53:44 DEBUG } 00:53:44 DEBUG ] 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG } 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG } 00:53:44 DEBUG } 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484424" 00:53:44 INFO [http_cmd]: http status code OK 00:53:44 DEBUG --- status code --- 00:53:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:44 DEBUG --- http response --- 00:53:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:44 DEBUG ----- output ----- 00:53:44 DEBUG { 00:53:44 DEBUG "status": "success", 00:53:44 DEBUG "data": { 00:53:44 DEBUG "resultType": "vector", 00:53:44 DEBUG "result": [ 00:53:44 DEBUG { 00:53:44 DEBUG "metric": { 00:53:44 DEBUG "pod": "ds-idrepo-0" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG [...] 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG "pod": "ds-idrepo-2" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 1713484424, 00:53:44 DEBUG "0" 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG } 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484424" 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484424" 00:53:44 INFO [http_cmd]: http status code OK 00:53:44 DEBUG --- status code --- 00:53:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:44 DEBUG --- http response --- 00:53:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:44 DEBUG ----- output ----- 00:53:44 DEBUG { 00:53:44 DEBUG "status": "success", 00:53:44 DEBUG "data": { 00:53:44 DEBUG "resultType": "vector", 00:53:44 DEBUG "result": [ 00:53:44 DEBUG { 00:53:44 DEBUG "metric": { 00:53:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG [...] 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 1713484424, 00:53:44 DEBUG "1901.0666666666666" 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG } 00:53:44 INFO [http_cmd]: http status code OK 00:53:44 DEBUG --- status code --- 00:53:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:44 DEBUG --- http response --- 00:53:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:53:44 DEBUG ----- output ----- 00:53:44 DEBUG { 00:53:44 DEBUG "status": "success", 00:53:44 DEBUG "data": { 00:53:44 DEBUG "resultType": "vector", 00:53:44 DEBUG "result": [ 00:53:44 DEBUG { 00:53:44 DEBUG "metric": { 00:53:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG [...] 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 1713484424, 00:53:44 DEBUG "0" 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG } 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484424" 00:53:44 INFO [http_cmd]: http status code OK 00:53:44 DEBUG --- status code --- 00:53:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:44 DEBUG --- http response --- 00:53:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:44 DEBUG ----- output ----- 00:53:44 DEBUG { 00:53:44 DEBUG "status": "success", 00:53:44 DEBUG "data": { 00:53:44 DEBUG "resultType": "vector", 00:53:44 DEBUG "result": [ 00:53:44 DEBUG { 00:53:44 DEBUG "metric": { 00:53:44 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG [...] 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG 00:53:44 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:53:44 DEBUG }, 00:53:44 DEBUG "value": [ 00:53:44 DEBUG 1713484424, 00:53:44 DEBUG "6.915564436150968" 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG ] 00:53:44 DEBUG } 00:53:44 DEBUG } 00:53:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484424" 00:53:45 INFO [http_cmd]: http status code OK 00:53:45 DEBUG --- status code --- 00:53:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:45 DEBUG --- http response --- 00:53:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:45 DEBUG ----- output ----- 00:53:45 DEBUG { 00:53:45 DEBUG "status": "success", 00:53:45 DEBUG "data": { 00:53:45 DEBUG "resultType": "vector", 00:53:45 DEBUG "result": [ 00:53:45 DEBUG { 00:53:45 DEBUG "metric": { 00:53:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:53:45 DEBUG }, 00:53:45 DEBUG "value": [ 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG [...] 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:53:45 DEBUG }, 00:53:45 DEBUG "value": [ 00:53:45 DEBUG 1713484424, 00:53:45 DEBUG "0" 00:53:45 DEBUG ] 00:53:45 DEBUG } 00:53:45 DEBUG ] 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 INFO 00:53:45 INFO [loop_until]: kubectl --namespace=xlou top pods 00:53:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:53:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484425" 00:53:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484425" 00:53:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484425" 00:53:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484425" 00:53:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484425" 00:53:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484425" 00:53:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484425" 00:53:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484425" 00:53:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484425" 00:53:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484425" 00:53:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484425" 00:53:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484425" 00:53:45 INFO [http_cmd]: http status code OK 00:53:45 INFO [http_cmd]: http status code OK 00:53:45 INFO [http_cmd]: http status code OK 00:53:45 INFO [http_cmd]: http status code OK 00:53:45 DEBUG --- status code --- 00:53:45 INFO [http_cmd]: http status code OK 00:53:45 INFO [http_cmd]: http status code OK 00:53:45 DEBUG --- status code --- 00:53:45 INFO [http_cmd]: http status code OK 00:53:45 DEBUG --- status code --- 00:53:45 DEBUG --- status code --- 00:53:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:45 DEBUG --- http response --- 00:53:45 INFO [http_cmd]: http status code OK 00:53:45 DEBUG --- status code --- 00:53:45 INFO [http_cmd]: http status code OK 00:53:45 DEBUG --- status code --- 00:53:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:45 DEBUG --- status code --- 00:53:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:45 INFO [http_cmd]: http status code OK 00:53:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:45 DEBUG --- status code --- 00:53:45 INFO [http_cmd]: http status code OK 00:53:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:45 DEBUG --- status code --- 00:53:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:45 DEBUG --- http response --- 00:53:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:45 DEBUG --- http response --- 00:53:45 DEBUG --- http response --- 00:53:45 DEBUG --- status code --- 00:53:45 DEBUG ----- output ----- 00:53:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:45 DEBUG --- status code --- 00:53:45 DEBUG --- http response --- 00:53:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:45 DEBUG --- http response --- 00:53:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:45 DEBUG --- http response --- 00:53:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:45 DEBUG { 00:53:45 DEBUG --- http response --- 00:53:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:45 INFO [http_cmd]: http status code OK 00:53:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:45 DEBUG --- http response --- 00:53:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:45 DEBUG ----- output ----- 00:53:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:53:45 DEBUG ----- output ----- 00:53:45 DEBUG ----- output ----- 00:53:45 DEBUG --- http response --- 00:53:45 DEBUG "status": "success", 00:53:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:45 DEBUG --- http response --- 00:53:45 DEBUG --- status code --- 00:53:45 DEBUG ----- output ----- 00:53:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:45 DEBUG ----- output ----- 00:53:45 DEBUG { 00:53:45 DEBUG ----- output ----- 00:53:45 DEBUG { 00:53:45 DEBUG { 00:53:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:53:45 DEBUG "data": { 00:53:45 DEBUG ----- output ----- 00:53:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:53:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:53:45 DEBUG { 00:53:45 DEBUG ----- output ----- 00:53:45 DEBUG { 00:53:45 DEBUG "status": "success", 00:53:45 DEBUG { 00:53:45 DEBUG "status": "success", 00:53:45 DEBUG "status": "success", 00:53:45 DEBUG ----- output ----- 00:53:45 DEBUG "resultType": "vector", 00:53:45 DEBUG { 00:53:45 DEBUG ----- output ----- 00:53:45 DEBUG --- http response --- 00:53:45 DEBUG "status": "success", 00:53:45 DEBUG { 00:53:45 DEBUG "status": "success", 00:53:45 DEBUG "data": { 00:53:45 DEBUG "status": "success", 00:53:45 DEBUG "data": { 00:53:45 DEBUG "data": { 00:53:45 DEBUG { 00:53:45 DEBUG "result": [ 00:53:45 DEBUG "status": "success", 00:53:45 DEBUG { 00:53:45 DEBUG "data": { 00:53:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:53:45 DEBUG "status": "success", 00:53:45 DEBUG "data": { 00:53:45 DEBUG "resultType": "vector", 00:53:45 DEBUG "data": { 00:53:45 DEBUG "resultType": "vector", 00:53:45 DEBUG "resultType": "vector", 00:53:45 DEBUG "status": "success", 00:53:45 DEBUG { 00:53:45 DEBUG "data": { 00:53:45 DEBUG "status": "success", 00:53:45 DEBUG "resultType": "vector", 00:53:45 DEBUG ----- output ----- 00:53:45 DEBUG "data": { 00:53:45 DEBUG "resultType": "vector", 00:53:45 DEBUG "result": [ 00:53:45 DEBUG "resultType": "vector", 00:53:45 DEBUG "result": [ 00:53:45 DEBUG "result": [ 00:53:45 DEBUG "data": { 00:53:45 DEBUG "metric": { 00:53:45 DEBUG "resultType": "vector", 00:53:45 DEBUG "data": { 00:53:45 DEBUG "result": [ 00:53:45 DEBUG { 00:53:45 DEBUG "resultType": "vector", 00:53:45 DEBUG "result": [ 00:53:45 DEBUG { 00:53:45 DEBUG "result": [ 00:53:45 DEBUG { 00:53:45 DEBUG { 00:53:45 DEBUG "resultType": "vector", 00:53:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:53:45 DEBUG "result": [ 00:53:45 DEBUG "resultType": "vector", 00:53:45 DEBUG { 00:53:45 DEBUG "status": "success", 00:53:45 DEBUG "result": [ 00:53:45 DEBUG { 00:53:45 DEBUG "metric": { 00:53:45 DEBUG "metric": { 00:53:45 DEBUG "metric": { 00:53:45 DEBUG "metric": { 00:53:45 DEBUG "result": [ 00:53:45 DEBUG }, 00:53:45 DEBUG { 00:53:45 DEBUG "result": [ 00:53:45 DEBUG "metric": { 00:53:45 DEBUG "data": { 00:53:45 DEBUG { 00:53:45 DEBUG { 00:53:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:53:45 DEBUG "pod": "ds-idrepo-0" 00:53:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:53:45 DEBUG "pod": "ds-idrepo-0" 00:53:45 DEBUG { 00:53:45 DEBUG "value": [ 00:53:45 DEBUG "metric": { 00:53:45 DEBUG { 00:53:45 DEBUG "instance": "10.142.0.107:9100" 00:53:45 DEBUG "resultType": "vector", 00:53:45 DEBUG "metric": { 00:53:45 DEBUG "metric": { 00:53:45 DEBUG }, 00:53:45 DEBUG }, 00:53:45 DEBUG }, 00:53:45 DEBUG }, 00:53:45 DEBUG "metric": { 00:53:45 DEBUG 00:53:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:53:45 DEBUG "metric": { 00:53:45 DEBUG }, 00:53:45 DEBUG "result": [ 00:53:45 DEBUG "instance": "10.142.0.107:9100" 00:53:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:53:45 DEBUG "value": [ 00:53:45 DEBUG "value": [ 00:53:45 DEBUG "value": [ 00:53:45 DEBUG "value": [ 00:53:45 DEBUG "pod": "ds-cts-0" 00:53:45 DEBUG 00:53:45 DEBUG }, 00:53:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:53:45 DEBUG "value": [ 00:53:45 DEBUG { 00:53:45 DEBUG }, 00:53:45 DEBUG }, 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG }, 00:53:45 DEBUG 00:53:45 DEBUG "value": [ 00:53:45 DEBUG }, 00:53:45 DEBUG 00:53:45 DEBUG "metric": { 00:53:45 DEBUG "value": [ 00:53:45 DEBUG "value": [ 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG "value": [ 00:53:45 DEBUG [...] 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG "pod": "ds-cts-0" 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG "value": [ 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG }, 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG [...] 00:53:45 DEBUG [...] 00:53:45 DEBUG [...] 00:53:45 DEBUG [...] 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG [...] 00:53:45 DEBUG [...] 00:53:45 DEBUG "value": [ 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG [...] 00:53:45 DEBUG [...] 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG [...] 00:53:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:53:45 DEBUG [...] 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG }, 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG [...] 00:53:45 DEBUG 00:53:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:53:45 DEBUG "pod": "ds-idrepo-2" 00:53:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:53:45 DEBUG "pod": "ds-idrepo-2" 00:53:45 DEBUG 00:53:45 DEBUG "value": [ 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG "instance": "10.142.0.43:9100" 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG }, 00:53:45 DEBUG }, 00:53:45 DEBUG }, 00:53:45 DEBUG }, 00:53:45 DEBUG 00:53:45 DEBUG 1713484425, 00:53:45 DEBUG 00:53:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:53:45 DEBUG }, 00:53:45 DEBUG 00:53:45 DEBUG 00:53:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:53:45 DEBUG "value": [ 00:53:45 DEBUG "value": [ 00:53:45 DEBUG 1713484425, 00:53:45 DEBUG "value": [ 00:53:45 DEBUG "pod": "ds-cts-2" 00:53:45 DEBUG "0" 00:53:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:53:45 DEBUG }, 00:53:45 DEBUG "value": [ 00:53:45 DEBUG "instance": "10.142.0.43:9100" 00:53:45 DEBUG 00:53:45 DEBUG }, 00:53:45 DEBUG 1713484425, 00:53:45 DEBUG "value": [ 00:53:45 DEBUG "0" 00:53:45 DEBUG 1713484425, 00:53:45 DEBUG }, 00:53:45 DEBUG ] 00:53:45 DEBUG }, 00:53:45 DEBUG "value": [ 00:53:45 DEBUG 1713484425, 00:53:45 DEBUG }, 00:53:45 DEBUG "pod": "ds-cts-2" 00:53:45 DEBUG "value": [ 00:53:45 DEBUG "1901.2333333333333" 00:53:45 DEBUG 1713484425, 00:53:45 DEBUG ] 00:53:45 DEBUG "0" 00:53:45 DEBUG "value": [ 00:53:45 DEBUG } 00:53:45 DEBUG "value": [ 00:53:45 DEBUG 1713484425, 00:53:45 DEBUG "0.37299999999999983" 00:53:45 DEBUG "value": [ 00:53:45 DEBUG }, 00:53:45 INFO [loop_until]: OK (rc = 0) 00:53:45 DEBUG 1713484425, 00:53:45 DEBUG ] 00:53:45 DEBUG "1051262976" 00:53:45 DEBUG } 00:53:45 DEBUG ] 00:53:45 DEBUG 1713484425, 00:53:45 DEBUG ] 00:53:45 DEBUG 1713484425, 00:53:45 DEBUG "20.076" 00:53:45 DEBUG ] 00:53:45 DEBUG 1713484425, 00:53:45 DEBUG "value": [ 00:53:45 DEBUG --- stdout --- 00:53:45 DEBUG "1150369.5" 00:53:45 DEBUG } 00:53:45 DEBUG ] 00:53:45 DEBUG ] 00:53:45 DEBUG } 00:53:45 DEBUG "0" 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG ] 00:53:45 DEBUG } 00:53:45 DEBUG "47056873.03333333" 00:53:45 DEBUG 1713484425, 00:53:45 DEBUG ] 00:53:45 DEBUG ] 00:53:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10497m 9400Mi am-6fd554f8f7-hvvkw 8540m 8881Mi am-6fd554f8f7-pmj4b 9139m 9025Mi ds-cts-0 9073m 12045Mi ds-cts-1 9309m 12046Mi ds-cts-2 8893m 12101Mi ds-idrepo-0 4807m 3257Mi ds-idrepo-1 4572m 3115Mi ds-idrepo-2 4624m 3038Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7000m 910Mi 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG ] 00:53:45 DEBUG ] 00:53:45 DEBUG "1901.5" 00:53:45 DEBUG } 00:53:45 DEBUG ] 00:53:45 DEBUG ] 00:53:45 DEBUG "336.43333333333334" 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG --- stderr --- 00:53:45 DEBUG ] 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG ] 00:53:45 DEBUG ] 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG ] 00:53:45 DEBUG ] 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG 00:53:45 DEBUG } 00:53:45 DEBUG ] 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG ] 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG ] 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG ] 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 DEBUG } 00:53:45 INFO 00:53:45 INFO [loop_until]: kubectl --namespace=xlou top node 00:53:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:53:45 INFO [loop_until]: OK (rc = 0) 00:53:45 DEBUG --- stdout --- 00:53:45 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5935Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4607m 15% 6650Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8383m 28% 11618Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3203Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9226m 30% 14483Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 166m 0% 5191Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10323m 34% 12193Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27566m 92% 3123Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9857m 33% 11252Mi 10% gke-xlou-cdm-ds-0f601751-j488 4967m 16% 6084Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8501m 28% 16014Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5246m 17% 6748Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8647m 28% 13992Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7050m 44% 4664Mi 7% 00:53:45 DEBUG --- stderr --- 00:53:45 DEBUG 00:54:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484484" 00:54:44 ERROR [http_cmd]: ERROR 00:54:44 ERROR --- status code --- 00:54:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:54:44 ERROR --- http response --- 00:54:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:54:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:54:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484484" 00:54:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484484" 00:54:44 INFO [http_cmd]: http status code OK 00:54:44 DEBUG --- status code --- 00:54:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:44 DEBUG --- http response --- 00:54:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:44 DEBUG ----- output ----- 00:54:44 DEBUG { 00:54:44 DEBUG "status": "success", 00:54:44 DEBUG "data": { 00:54:44 DEBUG "resultType": "vector", 00:54:44 DEBUG "result": [ 00:54:44 DEBUG { 00:54:44 DEBUG "metric": { 00:54:44 DEBUG "instance": "10.142.0.107:9100" 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG [...] 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG "instance": "10.142.0.43:9100" 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 1713484484, 00:54:44 DEBUG "0.9683000000000012" 00:54:44 DEBUG ] 00:54:44 DEBUG } 00:54:44 DEBUG ] 00:54:44 DEBUG } 00:54:44 DEBUG } 00:54:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484484" 00:54:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484484" 00:54:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484484" 00:54:44 INFO [http_cmd]: http status code OK 00:54:44 DEBUG --- status code --- 00:54:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:44 DEBUG --- http response --- 00:54:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:54:44 INFO [http_cmd]: http status code OK 00:54:44 INFO [http_cmd]: http status code OK 00:54:44 DEBUG --- status code --- 00:54:44 DEBUG --- status code --- 00:54:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:44 DEBUG --- http response --- 00:54:44 DEBUG --- http response --- 00:54:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:54:44 DEBUG ----- output ----- 00:54:44 DEBUG ----- output ----- 00:54:44 DEBUG { 00:54:44 DEBUG { 00:54:44 DEBUG "status": "success", 00:54:44 DEBUG "status": "success", 00:54:44 DEBUG "data": { 00:54:44 DEBUG "data": { 00:54:44 DEBUG "resultType": "vector", 00:54:44 DEBUG "resultType": "vector", 00:54:44 DEBUG "result": [ 00:54:44 DEBUG "result": [ 00:54:44 DEBUG { 00:54:44 DEBUG { 00:54:44 DEBUG "metric": { 00:54:44 DEBUG "metric": { 00:54:44 DEBUG "instance": "10.142.0.107:9100" 00:54:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:54:44 DEBUG }, 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG [...] 00:54:44 DEBUG [...] 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 INFO [http_cmd]: http status code OK 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG --- status code --- 00:54:44 DEBUG "instance": "10.142.0.43:9100" 00:54:44 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:54:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:44 DEBUG }, 00:54:44 DEBUG }, 00:54:44 DEBUG --- http response --- 00:54:44 DEBUG "value": [ 00:54:44 DEBUG "value": [ 00:54:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:44 DEBUG 1713484484, 00:54:44 DEBUG 1713484484, 00:54:44 DEBUG ----- output ----- 00:54:44 DEBUG "298217909248" 00:54:44 DEBUG "0" 00:54:44 DEBUG { 00:54:44 DEBUG ] 00:54:44 DEBUG ] 00:54:44 DEBUG "status": "success", 00:54:44 DEBUG } 00:54:44 DEBUG } 00:54:44 DEBUG "data": { 00:54:44 DEBUG ] 00:54:44 DEBUG ] 00:54:44 DEBUG "resultType": "vector", 00:54:44 DEBUG } 00:54:44 DEBUG } 00:54:44 DEBUG "result": [ 00:54:44 DEBUG } 00:54:44 DEBUG } 00:54:44 DEBUG { 00:54:44 DEBUG "metric": { 00:54:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG [...] 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 1713484484, 00:54:44 DEBUG "0" 00:54:44 DEBUG ] 00:54:44 DEBUG } 00:54:44 DEBUG ] 00:54:44 DEBUG } 00:54:44 DEBUG } 00:54:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484484" 00:54:44 INFO [http_cmd]: http status code OK 00:54:44 DEBUG --- status code --- 00:54:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:44 DEBUG --- http response --- 00:54:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:54:44 DEBUG ----- output ----- 00:54:44 DEBUG { 00:54:44 DEBUG "status": "success", 00:54:44 DEBUG "data": { 00:54:44 DEBUG "resultType": "vector", 00:54:44 DEBUG "result": [ 00:54:44 DEBUG { 00:54:44 DEBUG "metric": { 00:54:44 DEBUG "pod": "ds-idrepo-0" 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG [...] 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG "pod": "ds-cts-2" 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 1713484484, 00:54:44 DEBUG "0" 00:54:44 DEBUG ] 00:54:44 DEBUG } 00:54:44 DEBUG ] 00:54:44 DEBUG } 00:54:44 DEBUG } 00:54:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484484" 00:54:44 INFO [http_cmd]: http status code OK 00:54:44 DEBUG --- status code --- 00:54:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:44 DEBUG --- http response --- 00:54:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:44 DEBUG ----- output ----- 00:54:44 DEBUG { 00:54:44 DEBUG "status": "success", 00:54:44 DEBUG "data": { 00:54:44 DEBUG "resultType": "vector", 00:54:44 DEBUG "result": [ 00:54:44 DEBUG { 00:54:44 DEBUG "metric": { 00:54:44 DEBUG "instance": "10.142.0.107:9100" 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG [...] 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG "instance": "10.142.0.43:9100" 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 1713484484, 00:54:44 DEBUG "47331361.766666666" 00:54:44 DEBUG ] 00:54:44 DEBUG } 00:54:44 DEBUG ] 00:54:44 DEBUG } 00:54:44 DEBUG } 00:54:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484484" 00:54:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484484" 00:54:44 INFO [http_cmd]: http status code OK 00:54:44 DEBUG --- status code --- 00:54:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:44 DEBUG --- http response --- 00:54:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:54:44 INFO [http_cmd]: http status code OK 00:54:44 DEBUG ----- output ----- 00:54:44 DEBUG --- status code --- 00:54:44 DEBUG { 00:54:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:44 DEBUG "status": "success", 00:54:44 DEBUG --- http response --- 00:54:44 DEBUG "data": { 00:54:44 DEBUG "resultType": "vector", 00:54:44 DEBUG "result": [ 00:54:44 DEBUG { 00:54:44 DEBUG "metric": { 00:54:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG [...] 00:54:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:54:44 DEBUG 00:54:44 DEBUG ----- output ----- 00:54:44 DEBUG 00:54:44 DEBUG { 00:54:44 DEBUG 00:54:44 DEBUG "status": "success", 00:54:44 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:54:44 DEBUG "data": { 00:54:44 DEBUG }, 00:54:44 DEBUG "resultType": "vector", 00:54:44 DEBUG "value": [ 00:54:44 DEBUG "result": [ 00:54:44 DEBUG 1713484484, 00:54:44 DEBUG { 00:54:44 DEBUG "36776.35507987458" 00:54:44 DEBUG "metric": { 00:54:44 DEBUG ] 00:54:44 DEBUG "pod": "ds-cts-0" 00:54:44 DEBUG } 00:54:44 DEBUG }, 00:54:44 DEBUG ] 00:54:44 DEBUG "value": [ 00:54:44 DEBUG } 00:54:44 DEBUG 00:54:44 DEBUG } 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG [...] 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG "pod": "ds-cts-2" 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 1713484484, 00:54:44 DEBUG "1.2666666666666666" 00:54:44 DEBUG ] 00:54:44 DEBUG } 00:54:44 DEBUG ] 00:54:44 DEBUG } 00:54:44 DEBUG } 00:54:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484484" 00:54:44 INFO [http_cmd]: http status code OK 00:54:44 DEBUG --- status code --- 00:54:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:44 DEBUG --- http response --- 00:54:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:44 DEBUG ----- output ----- 00:54:44 DEBUG { 00:54:44 DEBUG "status": "success", 00:54:44 DEBUG "data": { 00:54:44 DEBUG "resultType": "vector", 00:54:44 DEBUG "result": [ 00:54:44 DEBUG { 00:54:44 DEBUG "metric": { 00:54:44 DEBUG "pod": "ds-idrepo-0" 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG [...] 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG "pod": "ds-idrepo-2" 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 1713484484, 00:54:44 DEBUG "0" 00:54:44 DEBUG ] 00:54:44 DEBUG } 00:54:44 DEBUG ] 00:54:44 DEBUG } 00:54:44 DEBUG } 00:54:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484484" 00:54:44 INFO [http_cmd]: http status code OK 00:54:44 DEBUG --- status code --- 00:54:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:44 DEBUG --- http response --- 00:54:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:44 DEBUG ----- output ----- 00:54:44 DEBUG { 00:54:44 DEBUG "status": "success", 00:54:44 DEBUG "data": { 00:54:44 DEBUG "resultType": "vector", 00:54:44 DEBUG "result": [ 00:54:44 DEBUG { 00:54:44 DEBUG "metric": { 00:54:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG [...] 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG 00:54:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:54:44 DEBUG }, 00:54:44 DEBUG "value": [ 00:54:44 DEBUG 1713484484, 00:54:44 DEBUG "1919.3333333333333" 00:54:44 DEBUG ] 00:54:44 DEBUG } 00:54:44 DEBUG ] 00:54:44 DEBUG } 00:54:44 DEBUG } 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484485" 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 DEBUG --- status code --- 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 DEBUG --- http response --- 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG { 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG "data": { 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG "result": [ 00:54:45 DEBUG { 00:54:45 DEBUG "metric": { 00:54:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:54:45 DEBUG }, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:54:45 DEBUG }, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG "8.45227966764418" 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484485" 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 DEBUG --- status code --- 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 DEBUG --- http response --- 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG { 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG "data": { 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG "result": [ 00:54:45 DEBUG { 00:54:45 DEBUG "metric": { 00:54:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:54:45 DEBUG }, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:54:45 DEBUG }, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG "6.893062379097186" 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484485" 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 DEBUG --- status code --- 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 DEBUG --- http response --- 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG { 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG "data": { 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG "result": [ 00:54:45 DEBUG { 00:54:45 DEBUG "metric": { 00:54:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:54:45 DEBUG }, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:54:45 DEBUG }, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG "0" 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 INFO 00:54:45 INFO [loop_until]: kubectl --namespace=xlou top pods 00:54:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484485" 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484485" 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484485" 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484485" 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484485" 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484485" 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484485" 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484485" 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484485" 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484485" 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484485" 00:54:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484485" 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 DEBUG --- status code --- 00:54:45 DEBUG --- status code --- 00:54:45 DEBUG --- status code --- 00:54:45 DEBUG --- status code --- 00:54:45 DEBUG --- status code --- 00:54:45 DEBUG --- status code --- 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 DEBUG --- http response --- 00:54:45 DEBUG --- status code --- 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 DEBUG --- status code --- 00:54:45 DEBUG --- http response --- 00:54:45 DEBUG --- http response --- 00:54:45 DEBUG --- http response --- 00:54:45 DEBUG --- http response --- 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 DEBUG --- http response --- 00:54:45 INFO [http_cmd]: http status code OK 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:54:45 DEBUG --- status code --- 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 INFO [loop_until]: OK (rc = 0) 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:45 DEBUG --- status code --- 00:54:45 DEBUG --- status code --- 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:45 DEBUG --- status code --- 00:54:45 DEBUG --- http response --- 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 DEBUG --- http response --- 00:54:45 DEBUG --- stdout --- 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:45 DEBUG { 00:54:45 DEBUG --- http response --- 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:54:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10323m 9378Mi am-6fd554f8f7-hvvkw 8446m 8881Mi am-6fd554f8f7-pmj4b 9125m 9026Mi ds-cts-0 8908m 12148Mi ds-cts-1 9357m 12348Mi ds-cts-2 8403m 12300Mi ds-idrepo-0 4734m 3257Mi ds-idrepo-1 4719m 3116Mi ds-idrepo-2 4607m 3038Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1214Mi lodemon-5d46798758-5h2st 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6732m 914Mi 00:54:45 DEBUG { 00:54:45 DEBUG { 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG { 00:54:45 DEBUG --- http response --- 00:54:45 DEBUG --- http response --- 00:54:45 DEBUG { 00:54:45 DEBUG --- http response --- 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG --- stderr --- 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG "data": { 00:54:45 DEBUG { 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:54:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:54:45 DEBUG { 00:54:45 DEBUG "data": { 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG { 00:54:45 DEBUG 00:54:45 DEBUG "data": { 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG "data": { 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG "data": { 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG ----- output ----- 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG "data": { 00:54:45 DEBUG { 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG "result": [ 00:54:45 DEBUG "data": { 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG { 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG { 00:54:45 DEBUG { 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG "data": { 00:54:45 DEBUG "result": [ 00:54:45 DEBUG { 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG "result": [ 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG "result": [ 00:54:45 DEBUG "status": "success", 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG "data": { 00:54:45 DEBUG "result": [ 00:54:45 DEBUG "data": { 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG { 00:54:45 DEBUG "metric": { 00:54:45 DEBUG "result": [ 00:54:45 DEBUG { 00:54:45 DEBUG "data": { 00:54:45 DEBUG { 00:54:45 DEBUG "data": { 00:54:45 DEBUG "result": [ 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG { 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG "result": [ 00:54:45 DEBUG "metric": { 00:54:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:54:45 DEBUG { 00:54:45 DEBUG "metric": { 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG "metric": { 00:54:45 DEBUG "resultType": "vector", 00:54:45 DEBUG { 00:54:45 DEBUG "result": [ 00:54:45 DEBUG "metric": { 00:54:45 DEBUG "result": [ 00:54:45 DEBUG { 00:54:45 DEBUG "pod": "ds-idrepo-0" 00:54:45 DEBUG }, 00:54:45 DEBUG "metric": { 00:54:45 DEBUG "instance": "10.142.0.107:9100" 00:54:45 DEBUG "result": [ 00:54:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:54:45 DEBUG "result": [ 00:54:45 DEBUG "metric": { 00:54:45 DEBUG { 00:54:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:54:45 DEBUG { 00:54:45 DEBUG "metric": { 00:54:45 DEBUG }, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG "pod": "ds-idrepo-0" 00:54:45 DEBUG }, 00:54:45 DEBUG { 00:54:45 DEBUG }, 00:54:45 DEBUG { 00:54:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:54:45 DEBUG "metric": { 00:54:45 DEBUG }, 00:54:45 DEBUG "metric": { 00:54:45 DEBUG "pod": "ds-cts-0" 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 00:54:45 DEBUG }, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG "metric": { 00:54:45 DEBUG "value": [ 00:54:45 DEBUG "metric": { 00:54:45 DEBUG }, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG "value": [ 00:54:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:54:45 DEBUG }, 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 00:54:45 DEBUG "instance": "10.142.0.107:9100" 00:54:45 DEBUG 00:54:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:54:45 DEBUG 00:54:45 DEBUG "pod": "ds-cts-0" 00:54:45 DEBUG 00:54:45 DEBUG }, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG }, 00:54:45 DEBUG 00:54:45 DEBUG }, 00:54:45 DEBUG 00:54:45 DEBUG }, 00:54:45 DEBUG 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG [...] 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG "pod": "ds-idrepo-2" 00:54:45 DEBUG }, 00:54:45 DEBUG 00:54:45 DEBUG "instance": "10.142.0.43:9100" 00:54:45 DEBUG 00:54:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:54:45 DEBUG 00:54:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:54:45 DEBUG 00:54:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:54:45 DEBUG 00:54:45 DEBUG 00:54:45 DEBUG }, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG "pod": "ds-idrepo-2" 00:54:45 DEBUG }, 00:54:45 DEBUG 00:54:45 DEBUG }, 00:54:45 DEBUG 00:54:45 DEBUG }, 00:54:45 DEBUG 00:54:45 DEBUG }, 00:54:45 DEBUG 00:54:45 DEBUG "pod": "ds-cts-2" 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG }, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 00:54:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:54:45 DEBUG 00:54:45 DEBUG "value": [ 00:54:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:54:45 DEBUG }, 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG "1919.3333333333333" 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG "instance": "10.142.0.43:9100" 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG }, 00:54:45 DEBUG "pod": "ds-cts-2" 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG }, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG "0" 00:54:45 DEBUG ] 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG "47337063" 00:54:45 DEBUG }, 00:54:45 DEBUG "20.076" 00:54:45 DEBUG ] 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG }, 00:54:45 DEBUG "1919.7333333333333" 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG "0" 00:54:45 DEBUG ] 00:54:45 DEBUG "value": [ 00:54:45 DEBUG } 00:54:45 DEBUG "value": [ 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG "value": [ 00:54:45 DEBUG ] 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG "0.00006666666666666667" 00:54:45 DEBUG } 00:54:45 DEBUG ] 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG "1264774.5" 00:54:45 DEBUG 1713484485, 00:54:45 DEBUG } 00:54:45 DEBUG "1056813056" 00:54:45 DEBUG ] 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG ] 00:54:45 DEBUG "1.0210000000000004" 00:54:45 DEBUG "0" 00:54:45 DEBUG } 00:54:45 DEBUG ] 00:54:45 DEBUG "2002.3" 00:54:45 DEBUG ] 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG ] 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG ] 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG ] 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 DEBUG } 00:54:45 INFO 00:54:45 INFO [loop_until]: kubectl --namespace=xlou top node 00:54:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:54:45 INFO [loop_until]: OK (rc = 0) 00:54:45 DEBUG --- stdout --- 00:54:45 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5928Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4620m 15% 6655Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8390m 28% 11613Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3209Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8887m 29% 14608Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 161m 0% 5215Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10341m 34% 12181Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27721m 92% 3130Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9942m 33% 11253Mi 10% gke-xlou-cdm-ds-0f601751-j488 4975m 16% 6080Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8807m 29% 16339Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5276m 17% 6740Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8264m 27% 14189Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 6807m 42% 4670Mi 7% 00:54:45 DEBUG --- stderr --- 00:54:45 DEBUG 00:55:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484544" 00:55:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484544" 00:55:44 ERROR [http_cmd]: ERROR 00:55:44 ERROR --- status code --- 00:55:44 INFO [http_cmd]: http status code OK 00:55:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:55:44 DEBUG --- status code --- 00:55:44 ERROR --- http response --- 00:55:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:44 DEBUG --- http response --- 00:55:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:55:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:55:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:44 DEBUG ----- output ----- 00:55:44 DEBUG { 00:55:44 DEBUG "status": "success", 00:55:44 DEBUG "data": { 00:55:44 DEBUG "resultType": "vector", 00:55:44 DEBUG "result": [ 00:55:44 DEBUG { 00:55:44 DEBUG "metric": { 00:55:44 DEBUG "instance": "10.142.0.107:9100" 00:55:44 DEBUG }, 00:55:44 DEBUG "value": [ 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG [...] 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG "instance": "10.142.0.43:9100" 00:55:44 DEBUG }, 00:55:44 DEBUG "value": [ 00:55:44 DEBUG 1713484544, 00:55:44 DEBUG "0.9741666666666674" 00:55:44 DEBUG ] 00:55:44 DEBUG } 00:55:44 DEBUG ] 00:55:44 DEBUG } 00:55:44 DEBUG } 00:55:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484544" 00:55:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484544" 00:55:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484544" 00:55:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484544" 00:55:44 INFO [http_cmd]: http status code OK 00:55:44 DEBUG --- status code --- 00:55:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:44 DEBUG --- http response --- 00:55:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:55:44 INFO [http_cmd]: http status code OK 00:55:44 DEBUG --- status code --- 00:55:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:44 DEBUG --- http response --- 00:55:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:44 INFO [http_cmd]: http status code OK 00:55:44 INFO [http_cmd]: http status code OK 00:55:44 DEBUG ----- output ----- 00:55:44 DEBUG --- status code --- 00:55:44 DEBUG --- status code --- 00:55:44 DEBUG { 00:55:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:44 DEBUG "status": "success", 00:55:44 DEBUG --- http response --- 00:55:44 DEBUG --- http response --- 00:55:44 DEBUG "data": { 00:55:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:44 DEBUG "resultType": "vector", 00:55:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:55:44 DEBUG ----- output ----- 00:55:44 DEBUG "result": [ 00:55:44 DEBUG ----- output ----- 00:55:44 DEBUG { 00:55:44 DEBUG { 00:55:44 DEBUG { 00:55:44 DEBUG "status": "success", 00:55:44 DEBUG "metric": { 00:55:44 DEBUG "status": "success", 00:55:44 DEBUG "data": { 00:55:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:55:44 DEBUG "data": { 00:55:44 DEBUG "resultType": "vector", 00:55:44 DEBUG }, 00:55:44 DEBUG "resultType": "vector", 00:55:44 DEBUG "result": [ 00:55:44 DEBUG "value": [ 00:55:44 DEBUG "result": [ 00:55:44 DEBUG { 00:55:44 DEBUG "metric": { 00:55:44 DEBUG { 00:55:44 DEBUG 00:55:44 DEBUG "instance": "10.142.0.107:9100" 00:55:44 DEBUG "metric": { 00:55:44 DEBUG 00:55:44 DEBUG }, 00:55:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:55:44 DEBUG 00:55:44 DEBUG "value": [ 00:55:44 DEBUG }, 00:55:44 DEBUG [...] 00:55:44 DEBUG 00:55:44 DEBUG "value": [ 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG [...] 00:55:44 DEBUG 00:55:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:55:44 DEBUG 00:55:44 DEBUG [...] 00:55:44 DEBUG }, 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG "value": [ 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 1713484544, 00:55:44 DEBUG "0" 00:55:44 DEBUG 00:55:44 DEBUG ] 00:55:44 DEBUG "instance": "10.142.0.43:9100" 00:55:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:55:44 DEBUG } 00:55:44 DEBUG }, 00:55:44 DEBUG }, 00:55:44 DEBUG ] 00:55:44 DEBUG "value": [ 00:55:44 DEBUG "value": [ 00:55:44 DEBUG } 00:55:44 DEBUG 1713484544, 00:55:44 DEBUG 1713484544, 00:55:44 DEBUG } 00:55:44 DEBUG "300507741184" 00:55:44 DEBUG "582.2110088482997" 00:55:44 DEBUG ] 00:55:44 DEBUG ] 00:55:44 DEBUG } 00:55:44 DEBUG } 00:55:44 DEBUG ] 00:55:44 DEBUG ] 00:55:44 DEBUG } 00:55:44 DEBUG } 00:55:44 DEBUG } 00:55:44 DEBUG } 00:55:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484544" 00:55:44 INFO [http_cmd]: http status code OK 00:55:44 DEBUG --- status code --- 00:55:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:44 DEBUG --- http response --- 00:55:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:55:44 DEBUG ----- output ----- 00:55:44 DEBUG { 00:55:44 DEBUG "status": "success", 00:55:44 DEBUG "data": { 00:55:44 DEBUG "resultType": "vector", 00:55:44 DEBUG "result": [ 00:55:44 DEBUG { 00:55:44 DEBUG "metric": { 00:55:44 DEBUG "pod": "ds-idrepo-0" 00:55:44 DEBUG }, 00:55:44 DEBUG "value": [ 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG [...] 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG "pod": "ds-cts-2" 00:55:44 DEBUG }, 00:55:44 DEBUG "value": [ 00:55:44 DEBUG 1713484544, 00:55:44 DEBUG "0" 00:55:44 DEBUG ] 00:55:44 DEBUG } 00:55:44 DEBUG ] 00:55:44 DEBUG } 00:55:44 DEBUG } 00:55:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484544" 00:55:44 INFO [http_cmd]: http status code OK 00:55:44 DEBUG --- status code --- 00:55:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:44 DEBUG --- http response --- 00:55:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:44 DEBUG ----- output ----- 00:55:44 DEBUG { 00:55:44 DEBUG "status": "success", 00:55:44 DEBUG "data": { 00:55:44 DEBUG "resultType": "vector", 00:55:44 DEBUG "result": [ 00:55:44 DEBUG { 00:55:44 DEBUG "metric": { 00:55:44 DEBUG "instance": "10.142.0.107:9100" 00:55:44 DEBUG }, 00:55:44 DEBUG "value": [ 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG [...] 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG "instance": "10.142.0.43:9100" 00:55:44 DEBUG }, 00:55:44 DEBUG "value": [ 00:55:44 DEBUG 1713484544, 00:55:44 DEBUG "47390990.266666666" 00:55:44 DEBUG ] 00:55:44 DEBUG } 00:55:44 DEBUG ] 00:55:44 DEBUG } 00:55:44 DEBUG } 00:55:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484544" 00:55:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484544" 00:55:44 INFO [http_cmd]: http status code OK 00:55:44 DEBUG --- status code --- 00:55:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:44 DEBUG --- http response --- 00:55:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:55:44 DEBUG ----- output ----- 00:55:44 DEBUG { 00:55:44 DEBUG "status": "success", 00:55:44 DEBUG "data": { 00:55:44 DEBUG "resultType": "vector", 00:55:44 DEBUG "result": [ 00:55:44 DEBUG { 00:55:44 DEBUG "metric": { 00:55:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:55:44 DEBUG }, 00:55:44 DEBUG "value": [ 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG [...] 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:55:44 DEBUG }, 00:55:44 DEBUG "value": [ 00:55:44 DEBUG 1713484544, 00:55:44 DEBUG "22124.01833623539" 00:55:44 DEBUG ] 00:55:44 DEBUG } 00:55:44 DEBUG ] 00:55:44 DEBUG } 00:55:44 DEBUG } 00:55:44 INFO [http_cmd]: http status code OK 00:55:44 DEBUG --- status code --- 00:55:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:44 DEBUG --- http response --- 00:55:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:55:44 DEBUG ----- output ----- 00:55:44 DEBUG { 00:55:44 DEBUG "status": "success", 00:55:44 DEBUG "data": { 00:55:44 DEBUG "resultType": "vector", 00:55:44 DEBUG "result": [ 00:55:44 DEBUG { 00:55:44 DEBUG "metric": { 00:55:44 DEBUG "pod": "ds-cts-0" 00:55:44 DEBUG }, 00:55:44 DEBUG "value": [ 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG [...] 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG 00:55:44 DEBUG "pod": "ds-cts-2" 00:55:44 DEBUG }, 00:55:44 DEBUG "value": [ 00:55:44 DEBUG 1713484544, 00:55:44 DEBUG "1.4" 00:55:44 DEBUG ] 00:55:44 DEBUG } 00:55:44 DEBUG ] 00:55:44 DEBUG } 00:55:44 DEBUG } 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484545" 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 DEBUG --- status code --- 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG { 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG "data": { 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG "result": [ 00:55:45 DEBUG { 00:55:45 DEBUG "metric": { 00:55:45 DEBUG "pod": "ds-idrepo-0" 00:55:45 DEBUG }, 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG [...] 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG "pod": "ds-idrepo-2" 00:55:45 DEBUG }, 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG "0" 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484545" 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 DEBUG --- status code --- 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG { 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG "data": { 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG "result": [ 00:55:45 DEBUG { 00:55:45 DEBUG "metric": { 00:55:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:55:45 DEBUG }, 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG [...] 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:55:45 DEBUG }, 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG "1918.9" 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484545" 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 DEBUG --- status code --- 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG { 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG "data": { 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG "result": [ 00:55:45 DEBUG { 00:55:45 DEBUG "metric": { 00:55:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:55:45 DEBUG }, 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG [...] 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:55:45 DEBUG }, 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG "9.161259585657938" 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484545" 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 DEBUG --- status code --- 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG { 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG "data": { 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG "result": [ 00:55:45 DEBUG { 00:55:45 DEBUG "metric": { 00:55:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:55:45 DEBUG }, 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG [...] 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:55:45 DEBUG }, 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG "6.905050690409247" 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484545" 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 DEBUG --- status code --- 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG { 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG "data": { 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG "result": [ 00:55:45 DEBUG { 00:55:45 DEBUG "metric": { 00:55:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:55:45 DEBUG }, 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG [...] 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:55:45 DEBUG }, 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG "0" 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 INFO 00:55:45 INFO [loop_until]: kubectl --namespace=xlou top pods 00:55:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484545" 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484545" 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484545" 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484545" 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484545" 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484545" 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484545" 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484545" 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484545" 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484545" 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484545" 00:55:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484545" 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 DEBUG --- status code --- 00:55:45 DEBUG --- status code --- 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 DEBUG --- status code --- 00:55:45 DEBUG --- status code --- 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG --- status code --- 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 DEBUG --- status code --- 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG --- http response --- 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG --- status code --- 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG --- status code --- 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:45 DEBUG --- status code --- 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG ----- output ----- 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 DEBUG --- http response --- 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 INFO [http_cmd]: http status code OK 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG { 00:55:45 DEBUG { 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG { 00:55:45 DEBUG --- status code --- 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:55:45 DEBUG --- status code --- 00:55:45 DEBUG --- status code --- 00:55:45 DEBUG { 00:55:45 DEBUG { 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:55:45 DEBUG { 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG "data": { 00:55:45 DEBUG "data": { 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG "data": { 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG { 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG --- http response --- 00:55:45 DEBUG "data": { 00:55:45 DEBUG "data": { 00:55:45 DEBUG { 00:55:45 DEBUG "data": { 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG { 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG "data": { 00:55:45 DEBUG "result": [ 00:55:45 DEBUG "result": [ 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG "result": [ 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG "data": { 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG ----- output ----- 00:55:45 DEBUG "result": [ 00:55:45 DEBUG "result": [ 00:55:45 INFO [loop_until]: OK (rc = 0) 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG { 00:55:45 DEBUG { 00:55:45 DEBUG "data": { 00:55:45 DEBUG { 00:55:45 DEBUG { 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG { 00:55:45 DEBUG { 00:55:45 DEBUG { 00:55:45 DEBUG { 00:55:45 DEBUG --- stdout --- 00:55:45 DEBUG "result": [ 00:55:45 DEBUG "result": [ 00:55:45 DEBUG "metric": { 00:55:45 DEBUG "metric": { 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG "metric": { 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG "result": [ 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG "status": "success", 00:55:45 DEBUG "metric": { 00:55:45 DEBUG "metric": { 00:55:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10503m 9378Mi am-6fd554f8f7-hvvkw 8413m 8881Mi am-6fd554f8f7-pmj4b 9127m 9026Mi ds-cts-0 8624m 12260Mi ds-cts-1 9436m 12389Mi ds-cts-2 8824m 12320Mi ds-idrepo-0 4713m 3257Mi ds-idrepo-1 4685m 3123Mi ds-idrepo-2 4591m 3038Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1171Mi idm-b85f9d978-wnhxb 3m 1214Mi lodemon-5d46798758-5h2st 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6894m 916Mi 00:55:45 DEBUG { 00:55:45 DEBUG { 00:55:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:55:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:55:45 DEBUG "result": [ 00:55:45 DEBUG "pod": "ds-idrepo-0" 00:55:45 DEBUG "data": { 00:55:45 DEBUG { 00:55:45 DEBUG "data": { 00:55:45 DEBUG "data": { 00:55:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:55:45 DEBUG "instance": "10.142.0.107:9100" 00:55:45 DEBUG --- stderr --- 00:55:45 DEBUG "metric": { 00:55:45 DEBUG "metric": { 00:55:45 DEBUG }, 00:55:45 DEBUG }, 00:55:45 DEBUG { 00:55:45 DEBUG }, 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG "metric": { 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG "resultType": "vector", 00:55:45 DEBUG }, 00:55:45 DEBUG }, 00:55:45 DEBUG 00:55:45 DEBUG "pod": "ds-idrepo-0" 00:55:45 DEBUG "pod": "ds-cts-0" 00:55:45 DEBUG "value": [ 00:55:45 DEBUG "value": [ 00:55:45 DEBUG "metric": { 00:55:45 DEBUG "value": [ 00:55:45 DEBUG "result": [ 00:55:45 DEBUG "pod": "ds-cts-0" 00:55:45 DEBUG "result": [ 00:55:45 DEBUG "result": [ 00:55:45 DEBUG "value": [ 00:55:45 DEBUG "value": [ 00:55:45 DEBUG }, 00:55:45 DEBUG }, 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:55:45 DEBUG 00:55:45 DEBUG { 00:55:45 DEBUG }, 00:55:45 DEBUG { 00:55:45 DEBUG { 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG "value": [ 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG }, 00:55:45 DEBUG 00:55:45 DEBUG "metric": { 00:55:45 DEBUG "value": [ 00:55:45 DEBUG "metric": { 00:55:45 DEBUG "metric": { 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 00:55:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:55:45 DEBUG 00:55:45 DEBUG "instance": "10.142.0.107:9100" 00:55:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG [...] 00:55:45 DEBUG 00:55:45 DEBUG [...] 00:55:45 DEBUG }, 00:55:45 DEBUG 00:55:45 DEBUG }, 00:55:45 DEBUG }, 00:55:45 DEBUG [...] 00:55:45 DEBUG [...] 00:55:45 DEBUG 00:55:45 DEBUG [...] 00:55:45 DEBUG [...] 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 00:55:45 DEBUG "value": [ 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG [...] 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG [...] 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG [...] 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG "instance": "10.142.0.43:9100" 00:55:45 DEBUG }, 00:55:45 DEBUG 00:55:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:55:45 DEBUG 00:55:45 DEBUG "pod": "ds-idrepo-2" 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG [...] 00:55:45 DEBUG 00:55:45 DEBUG 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 00:55:45 DEBUG }, 00:55:45 DEBUG 00:55:45 DEBUG }, 00:55:45 DEBUG [...] 00:55:45 DEBUG 00:55:45 DEBUG [...] 00:55:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:55:45 DEBUG 00:55:45 DEBUG "pod": "ds-cts-2" 00:55:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG "pod": "ds-idrepo-2" 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 00:55:45 DEBUG "pod": "ds-cts-2" 00:55:45 DEBUG 00:55:45 DEBUG }, 00:55:45 DEBUG 00:55:45 DEBUG }, 00:55:45 DEBUG }, 00:55:45 DEBUG "47396596.8" 00:55:45 DEBUG }, 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG 00:55:45 DEBUG }, 00:55:45 DEBUG 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 00:55:45 DEBUG "value": [ 00:55:45 DEBUG "value": [ 00:55:45 DEBUG ] 00:55:45 DEBUG "value": [ 00:55:45 DEBUG "20.076" 00:55:45 DEBUG }, 00:55:45 DEBUG "0" 00:55:45 DEBUG 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:55:45 DEBUG }, 00:55:45 DEBUG "value": [ 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG ] 00:55:45 DEBUG "value": [ 00:55:45 DEBUG ] 00:55:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG "instance": "10.142.0.43:9100" 00:55:45 DEBUG "1918.8333333333333" 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG } 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG "0" 00:55:45 DEBUG "0" 00:55:45 DEBUG } 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG } 00:55:45 DEBUG }, 00:55:45 DEBUG "1813.1" 00:55:45 DEBUG }, 00:55:45 DEBUG ] 00:55:45 DEBUG "1918.8" 00:55:45 DEBUG ] 00:55:45 DEBUG "0" 00:55:45 DEBUG ] 00:55:45 DEBUG ] 00:55:45 DEBUG ] 00:55:45 DEBUG "1058627584" 00:55:45 DEBUG ] 00:55:45 DEBUG "value": [ 00:55:45 DEBUG ] 00:55:45 DEBUG "value": [ 00:55:45 DEBUG } 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG } 00:55:45 DEBUG 1713484545, 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG ] 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG "1379583.5" 00:55:45 DEBUG ] 00:55:45 DEBUG "1.395333333333333" 00:55:45 DEBUG } 00:55:45 DEBUG ] 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG ] 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG ] 00:55:45 DEBUG ] 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 DEBUG } 00:55:45 INFO 00:55:45 INFO [loop_until]: kubectl --namespace=xlou top node 00:55:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:55:45 INFO [loop_until]: OK (rc = 0) 00:55:45 DEBUG --- stdout --- 00:55:45 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5921Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4626m 15% 6661Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8412m 28% 11619Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3224Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8780m 29% 14709Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 5226Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10447m 34% 12187Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27984m 93% 3125Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9787m 32% 11249Mi 10% gke-xlou-cdm-ds-0f601751-j488 4993m 16% 6089Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9596m 32% 16390Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5208m 17% 6719Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8166m 27% 14207Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7065m 44% 4672Mi 7% 00:55:45 DEBUG --- stderr --- 00:55:45 DEBUG 00:56:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484604" 00:56:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484604" 00:56:44 ERROR [http_cmd]: ERROR 00:56:44 INFO [http_cmd]: http status code OK 00:56:44 ERROR --- status code --- 00:56:44 DEBUG --- status code --- 00:56:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:56:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:44 ERROR --- http response --- 00:56:44 DEBUG --- http response --- 00:56:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:56:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:56:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:56:44 DEBUG ----- output ----- 00:56:44 DEBUG { 00:56:44 DEBUG "status": "success", 00:56:44 DEBUG "data": { 00:56:44 DEBUG "resultType": "vector", 00:56:44 DEBUG "result": [ 00:56:44 DEBUG { 00:56:44 DEBUG "metric": { 00:56:44 DEBUG "instance": "10.142.0.107:9100" 00:56:44 DEBUG }, 00:56:44 DEBUG "value": [ 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG [...] 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG "instance": "10.142.0.43:9100" 00:56:44 DEBUG }, 00:56:44 DEBUG "value": [ 00:56:44 DEBUG 1713484604, 00:56:44 DEBUG "0.9729000000000032" 00:56:44 DEBUG ] 00:56:44 DEBUG } 00:56:44 DEBUG ] 00:56:44 DEBUG } 00:56:44 DEBUG } 00:56:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484604" 00:56:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484604" 00:56:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484604" 00:56:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484604" 00:56:44 INFO [http_cmd]: http status code OK 00:56:44 DEBUG --- status code --- 00:56:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:44 DEBUG --- http response --- 00:56:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:56:44 INFO [http_cmd]: http status code OK 00:56:44 DEBUG --- status code --- 00:56:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:44 DEBUG --- http response --- 00:56:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:56:44 DEBUG ----- output ----- 00:56:44 DEBUG { 00:56:44 DEBUG "status": "success", 00:56:44 DEBUG "data": { 00:56:44 DEBUG "resultType": "vector", 00:56:44 DEBUG "result": [ 00:56:44 DEBUG { 00:56:44 DEBUG "metric": { 00:56:44 DEBUG "instance": "10.142.0.107:9100" 00:56:44 DEBUG }, 00:56:44 DEBUG "value": [ 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG [...] 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG "instance": "10.142.0.43:9100" 00:56:44 DEBUG }, 00:56:44 DEBUG "value": [ 00:56:44 DEBUG 1713484604, 00:56:44 DEBUG "302875681792" 00:56:44 DEBUG ] 00:56:44 DEBUG } 00:56:44 DEBUG ] 00:56:44 DEBUG } 00:56:44 DEBUG } 00:56:44 INFO [http_cmd]: http status code OK 00:56:44 INFO [http_cmd]: http status code OK 00:56:44 DEBUG --- status code --- 00:56:44 DEBUG --- status code --- 00:56:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:44 DEBUG --- http response --- 00:56:44 DEBUG --- http response --- 00:56:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:56:44 DEBUG ----- output ----- 00:56:44 DEBUG ----- output ----- 00:56:44 DEBUG { 00:56:44 DEBUG { 00:56:44 DEBUG "status": "success", 00:56:44 DEBUG "status": "success", 00:56:44 DEBUG "data": { 00:56:44 DEBUG "data": { 00:56:44 DEBUG "resultType": "vector", 00:56:44 DEBUG "resultType": "vector", 00:56:44 DEBUG "result": [ 00:56:44 DEBUG "result": [ 00:56:44 DEBUG { 00:56:44 DEBUG { 00:56:44 DEBUG "metric": { 00:56:44 DEBUG "metric": { 00:56:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:56:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:56:44 DEBUG }, 00:56:44 DEBUG }, 00:56:44 DEBUG "value": [ 00:56:44 DEBUG "value": [ 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG [...] 00:56:44 DEBUG [...] 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:56:44 DEBUG "pod": "ds-cts-2" 00:56:44 DEBUG }, 00:56:44 DEBUG }, 00:56:44 DEBUG "value": [ 00:56:44 DEBUG "value": [ 00:56:44 DEBUG 1713484604, 00:56:44 DEBUG 1713484604, 00:56:44 DEBUG "0" 00:56:44 DEBUG "20479490.676203575" 00:56:44 DEBUG ] 00:56:44 DEBUG ] 00:56:44 DEBUG } 00:56:44 DEBUG } 00:56:44 DEBUG ] 00:56:44 DEBUG ] 00:56:44 DEBUG } 00:56:44 DEBUG } 00:56:44 DEBUG } 00:56:44 DEBUG } 00:56:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484604" 00:56:44 INFO [http_cmd]: http status code OK 00:56:44 DEBUG --- status code --- 00:56:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:44 DEBUG --- http response --- 00:56:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:56:44 DEBUG ----- output ----- 00:56:44 DEBUG { 00:56:44 DEBUG "status": "success", 00:56:44 DEBUG "data": { 00:56:44 DEBUG "resultType": "vector", 00:56:44 DEBUG "result": [ 00:56:44 DEBUG { 00:56:44 DEBUG "metric": { 00:56:44 DEBUG "pod": "ds-idrepo-0" 00:56:44 DEBUG }, 00:56:44 DEBUG "value": [ 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG [...] 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG 00:56:44 DEBUG "pod": "ds-cts-2" 00:56:44 DEBUG }, 00:56:44 DEBUG "value": [ 00:56:44 DEBUG 1713484604, 00:56:44 DEBUG "0" 00:56:44 DEBUG ] 00:56:44 DEBUG } 00:56:44 DEBUG ] 00:56:44 DEBUG } 00:56:44 DEBUG } 00:56:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484604" 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG { 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "data": { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG "result": [ 00:56:45 DEBUG { 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "instance": "10.142.0.107:9100" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "instance": "10.142.0.43:9100" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 1713484604, 00:56:45 DEBUG "47416429.16666667" 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG { 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "data": { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG "result": [ 00:56:45 DEBUG { 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG "303695898.6997829" 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG { 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "data": { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG "result": [ 00:56:45 DEBUG { 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "pod": "ds-cts-0" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "pod": "ds-cts-2" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG "1.2" 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG { 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "data": { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG "result": [ 00:56:45 DEBUG { 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "pod": "ds-idrepo-0" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "pod": "ds-idrepo-2" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG "0" 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG { 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "data": { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG "result": [ 00:56:45 DEBUG { 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG "1919.5666666666666" 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG { 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "data": { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG "result": [ 00:56:45 DEBUG { 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG "5.753727178689818" 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484605" 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG { 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "data": { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG "result": [ 00:56:45 DEBUG { 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG "6.946796999041417" 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG { 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "data": { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG "result": [ 00:56:45 DEBUG { 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG "0" 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 INFO 00:56:45 INFO [loop_until]: kubectl --namespace=xlou top pods 00:56:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484605" 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484605" 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484605" 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484605" 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG --- status code --- 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG ----- output ----- 00:56:45 INFO [http_cmd]: http status code OK 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG { 00:56:45 DEBUG { 00:56:45 DEBUG --- status code --- 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:45 DEBUG { 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG { 00:56:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG "data": { 00:56:45 DEBUG "data": { 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG { 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:56:45 DEBUG --- http response --- 00:56:45 DEBUG { 00:56:45 DEBUG "data": { 00:56:45 DEBUG { 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "data": { 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG { 00:56:45 DEBUG "data": { 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "result": [ 00:56:45 DEBUG "result": [ 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG "data": { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG { 00:56:45 DEBUG "data": { 00:56:45 DEBUG ----- output ----- 00:56:45 DEBUG "result": [ 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "data": { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG { 00:56:45 DEBUG { 00:56:45 DEBUG { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG "result": [ 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG { 00:56:45 DEBUG { 00:56:45 DEBUG "data": { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG "result": [ 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "result": [ 00:56:45 DEBUG { 00:56:45 DEBUG "data": { 00:56:45 INFO [loop_until]: OK (rc = 0) 00:56:45 DEBUG "result": [ 00:56:45 DEBUG "status": "success", 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG "result": [ 00:56:45 DEBUG { 00:56:45 DEBUG "pod": "ds-idrepo-0" 00:56:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:56:45 DEBUG "data": { 00:56:45 DEBUG { 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG --- stdout --- 00:56:45 DEBUG { 00:56:45 DEBUG "data": { 00:56:45 DEBUG "instance": "10.142.0.107:9100" 00:56:45 DEBUG "result": [ 00:56:45 DEBUG { 00:56:45 DEBUG "metric": { 00:56:45 DEBUG }, 00:56:45 DEBUG }, 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "pod": "ds-cts-0" 00:56:45 DEBUG "result": [ 00:56:45 DEBUG "metric": { 00:56:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10465m 9380Mi am-6fd554f8f7-hvvkw 8292m 8873Mi am-6fd554f8f7-pmj4b 9013m 9026Mi ds-cts-0 9292m 12303Mi ds-cts-1 8344m 12275Mi ds-cts-2 9297m 12323Mi ds-idrepo-0 4842m 3258Mi ds-idrepo-1 4788m 3106Mi ds-idrepo-2 4592m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7042m 917Mi 00:56:45 DEBUG "resultType": "vector", 00:56:45 DEBUG }, 00:56:45 DEBUG { 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "instance": "10.142.0.107:9100" 00:56:45 DEBUG "value": [ 00:56:45 DEBUG "value": [ 00:56:45 DEBUG "result": [ 00:56:45 DEBUG "pod": "ds-cts-0" 00:56:45 DEBUG }, 00:56:45 DEBUG { 00:56:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:56:45 DEBUG --- stderr --- 00:56:45 DEBUG "result": [ 00:56:45 DEBUG "value": [ 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:56:45 DEBUG }, 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG { 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG "metric": { 00:56:45 DEBUG }, 00:56:45 DEBUG 00:56:45 DEBUG { 00:56:45 DEBUG 00:56:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "metric": { 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:56:45 DEBUG "value": [ 00:56:45 DEBUG "metric": { 00:56:45 DEBUG 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "pod": "ds-idrepo-0" 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG }, 00:56:45 DEBUG 00:56:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:56:45 DEBUG 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG [...] 00:56:45 DEBUG }, 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG }, 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "pod": "ds-idrepo-2" 00:56:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "instance": "10.142.0.43:9100" 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG }, 00:56:45 DEBUG }, 00:56:45 DEBUG [...] 00:56:45 DEBUG 00:56:45 DEBUG "pod": "ds-cts-2" 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG [...] 00:56:45 DEBUG }, 00:56:45 DEBUG 00:56:45 DEBUG 00:56:45 DEBUG "instance": "10.142.0.43:9100" 00:56:45 DEBUG "value": [ 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG "pod": "ds-cts-2" 00:56:45 DEBUG }, 00:56:45 DEBUG 00:56:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:56:45 DEBUG 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:56:45 DEBUG }, 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG }, 00:56:45 DEBUG 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG "0" 00:56:45 DEBUG "20.076" 00:56:45 DEBUG 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 00:56:45 DEBUG "47422004.03333333" 00:56:45 DEBUG }, 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG ] 00:56:45 DEBUG ] 00:56:45 DEBUG "pod": "ds-idrepo-2" 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG "2210.9333333333334" 00:56:45 DEBUG }, 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:56:45 DEBUG ] 00:56:45 DEBUG "value": [ 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG "1.2833333333333337" 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG }, 00:56:45 DEBUG "0.00011964666666666667" 00:56:45 DEBUG ] 00:56:45 DEBUG "value": [ 00:56:45 DEBUG "1494551.5" 00:56:45 DEBUG }, 00:56:45 DEBUG } 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG "1919.5666666666666" 00:56:45 DEBUG ] 00:56:45 DEBUG ] 00:56:45 DEBUG ] 00:56:45 DEBUG "value": [ 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG ] 00:56:45 DEBUG "value": [ 00:56:45 DEBUG ] 00:56:45 DEBUG "1919.3999999999999" 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG "0" 00:56:45 DEBUG } 00:56:45 DEBUG 1713484605, 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG "0" 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG ] 00:56:45 DEBUG "1060638720" 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG ] 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 DEBUG } 00:56:45 INFO 00:56:45 INFO [loop_until]: kubectl --namespace=xlou top node 00:56:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:56:45 INFO [loop_until]: OK (rc = 0) 00:56:45 DEBUG --- stdout --- 00:56:45 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5923Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4632m 15% 6657Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8497m 28% 11598Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3206Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9304m 31% 14790Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 180m 0% 5203Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10405m 34% 12177Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28022m 93% 3123Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9751m 32% 11248Mi 10% gke-xlou-cdm-ds-0f601751-j488 5002m 16% 6079Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8354m 27% 16292Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5229m 17% 6739Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8716m 29% 14235Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7087m 44% 4672Mi 7% 00:56:45 DEBUG --- stderr --- 00:56:45 DEBUG 00:57:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484664" 00:57:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484664" 00:57:44 INFO [http_cmd]: http status code OK 00:57:44 DEBUG --- status code --- 00:57:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:44 DEBUG --- http response --- 00:57:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:44 DEBUG ----- output ----- 00:57:44 DEBUG { 00:57:44 DEBUG "status": "success", 00:57:44 DEBUG "data": { 00:57:44 DEBUG "resultType": "vector", 00:57:44 DEBUG "result": [ 00:57:44 DEBUG { 00:57:44 DEBUG "metric": { 00:57:44 DEBUG "instance": "10.142.0.107:9100" 00:57:44 DEBUG }, 00:57:44 DEBUG "value": [ 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG [...] 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG "instance": "10.142.0.43:9100" 00:57:44 DEBUG }, 00:57:44 DEBUG "value": [ 00:57:44 DEBUG 1713484664, 00:57:44 DEBUG "0.9605999999999976" 00:57:44 DEBUG ] 00:57:44 DEBUG } 00:57:44 DEBUG ] 00:57:44 DEBUG } 00:57:44 DEBUG } 00:57:44 ERROR [http_cmd]: ERROR 00:57:44 ERROR --- status code --- 00:57:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:57:44 ERROR --- http response --- 00:57:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:57:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:57:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484664" 00:57:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484664" 00:57:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484664" 00:57:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484664" 00:57:44 INFO [http_cmd]: http status code OK 00:57:44 DEBUG --- status code --- 00:57:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:44 DEBUG --- http response --- 00:57:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:57:44 INFO [http_cmd]: http status code OK 00:57:44 INFO [http_cmd]: http status code OK 00:57:44 DEBUG --- status code --- 00:57:44 DEBUG --- status code --- 00:57:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:44 DEBUG --- http response --- 00:57:44 DEBUG --- http response --- 00:57:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:57:44 DEBUG ----- output ----- 00:57:44 DEBUG ----- output ----- 00:57:44 DEBUG { 00:57:44 DEBUG { 00:57:44 DEBUG "status": "success", 00:57:44 DEBUG "status": "success", 00:57:44 DEBUG "data": { 00:57:44 DEBUG "data": { 00:57:44 DEBUG "resultType": "vector", 00:57:44 DEBUG "resultType": "vector", 00:57:44 DEBUG "result": [ 00:57:44 DEBUG { 00:57:44 DEBUG "metric": { 00:57:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:57:44 DEBUG }, 00:57:44 DEBUG "value": [ 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG [...] 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:57:44 DEBUG }, 00:57:44 DEBUG "value": [ 00:57:44 DEBUG 1713484664, 00:57:44 DEBUG "0" 00:57:44 DEBUG ] 00:57:44 DEBUG } 00:57:44 DEBUG ] 00:57:44 DEBUG "result": [ 00:57:44 DEBUG } 00:57:44 DEBUG { 00:57:44 DEBUG } 00:57:44 DEBUG "metric": { 00:57:44 DEBUG "instance": "10.142.0.107:9100" 00:57:44 DEBUG }, 00:57:44 DEBUG "value": [ 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG [...] 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG "instance": "10.142.0.43:9100" 00:57:44 DEBUG }, 00:57:44 DEBUG "value": [ 00:57:44 DEBUG 1713484664, 00:57:44 DEBUG "305194365952" 00:57:44 DEBUG ] 00:57:44 DEBUG } 00:57:44 DEBUG ] 00:57:44 DEBUG } 00:57:44 DEBUG } 00:57:44 INFO [http_cmd]: http status code OK 00:57:44 DEBUG --- status code --- 00:57:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:44 DEBUG --- http response --- 00:57:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:44 DEBUG ----- output ----- 00:57:44 DEBUG { 00:57:44 DEBUG "status": "success", 00:57:44 DEBUG "data": { 00:57:44 DEBUG "resultType": "vector", 00:57:44 DEBUG "result": [ 00:57:44 DEBUG { 00:57:44 DEBUG "metric": { 00:57:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:57:44 DEBUG }, 00:57:44 DEBUG "value": [ 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG [...] 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:57:44 DEBUG }, 00:57:44 DEBUG "value": [ 00:57:44 DEBUG 1713484664, 00:57:44 DEBUG "0" 00:57:44 DEBUG ] 00:57:44 DEBUG } 00:57:44 DEBUG ] 00:57:44 DEBUG } 00:57:44 DEBUG } 00:57:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484664" 00:57:44 INFO [http_cmd]: http status code OK 00:57:44 DEBUG --- status code --- 00:57:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:44 DEBUG --- http response --- 00:57:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:57:44 DEBUG ----- output ----- 00:57:44 DEBUG { 00:57:44 DEBUG "status": "success", 00:57:44 DEBUG "data": { 00:57:44 DEBUG "resultType": "vector", 00:57:44 DEBUG "result": [ 00:57:44 DEBUG { 00:57:44 DEBUG "metric": { 00:57:44 DEBUG "pod": "ds-idrepo-0" 00:57:44 DEBUG }, 00:57:44 DEBUG "value": [ 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG [...] 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG 00:57:44 DEBUG "pod": "ds-cts-2" 00:57:44 DEBUG }, 00:57:44 DEBUG "value": [ 00:57:44 DEBUG 1713484664, 00:57:44 DEBUG "0" 00:57:44 DEBUG ] 00:57:44 DEBUG } 00:57:44 DEBUG ] 00:57:44 DEBUG } 00:57:44 DEBUG } 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484665" 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG { 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "data": { 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "result": [ 00:57:45 DEBUG { 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "instance": "10.142.0.107:9100" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "instance": "10.142.0.43:9100" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG "47307044.43333334" 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG { 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "data": { 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "result": [ 00:57:45 DEBUG { 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG "36509.15488784991" 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG { 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "data": { 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "result": [ 00:57:45 DEBUG { 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "pod": "ds-cts-0" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "pod": "ds-cts-2" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG "0.9333333333333333" 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG { 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "data": { 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "result": [ 00:57:45 DEBUG { 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "pod": "ds-idrepo-0" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "pod": "ds-idrepo-2" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG "0" 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG { 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "data": { 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "result": [ 00:57:45 DEBUG { 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG "1922.5333333333333" 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG { 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "data": { 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "result": [ 00:57:45 DEBUG { 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG "13.907028119561776" 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484665" 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG { 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "data": { 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "result": [ 00:57:45 DEBUG { 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG "6.93609832512364" 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG { 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "data": { 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "result": [ 00:57:45 DEBUG { 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG "0" 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 INFO 00:57:45 INFO [loop_until]: kubectl --namespace=xlou top pods 00:57:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484665" 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484665" 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484665" 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484665" 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG --- status code --- 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG --- status code --- 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG --- status code --- 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG --- http response --- 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG --- status code --- 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 INFO [http_cmd]: http status code OK 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG { 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG --- status code --- 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG { 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:45 DEBUG { 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG { 00:57:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "data": { 00:57:45 DEBUG { 00:57:45 DEBUG { 00:57:45 DEBUG { 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG --- http response --- 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG "data": { 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:57:45 DEBUG { 00:57:45 DEBUG { 00:57:45 DEBUG "data": { 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "data": { 00:57:45 DEBUG { 00:57:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "result": [ 00:57:45 DEBUG "data": { 00:57:45 DEBUG "data": { 00:57:45 DEBUG "data": { 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG ----- output ----- 00:57:45 DEBUG "result": [ 00:57:45 DEBUG { 00:57:45 DEBUG "data": { 00:57:45 DEBUG "data": { 00:57:45 DEBUG "result": [ 00:57:45 DEBUG { 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "result": [ 00:57:45 DEBUG "data": { 00:57:45 DEBUG { 00:57:45 DEBUG { 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG { 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "result": [ 00:57:45 DEBUG "result": [ 00:57:45 DEBUG "result": [ 00:57:45 DEBUG { 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG "status": "success", 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "data": { 00:57:45 DEBUG "result": [ 00:57:45 DEBUG "result": [ 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "instance": "10.142.0.107:9100" 00:57:45 DEBUG { 00:57:45 DEBUG "metric": { 00:57:45 DEBUG { 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "result": [ 00:57:45 DEBUG "data": { 00:57:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:57:45 DEBUG "resultType": "vector", 00:57:45 INFO [loop_until]: OK (rc = 0) 00:57:45 DEBUG { 00:57:45 DEBUG { 00:57:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:57:45 DEBUG }, 00:57:45 DEBUG "pod": "ds-idrepo-0" 00:57:45 DEBUG { 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "pod": "ds-idrepo-0" 00:57:45 DEBUG { 00:57:45 DEBUG "resultType": "vector", 00:57:45 DEBUG }, 00:57:45 DEBUG "result": [ 00:57:45 DEBUG --- stdout --- 00:57:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10251m 9380Mi am-6fd554f8f7-hvvkw 8565m 8873Mi am-6fd554f8f7-pmj4b 9152m 9027Mi ds-cts-0 9260m 12304Mi ds-cts-1 8472m 12291Mi ds-cts-2 8383m 12355Mi ds-idrepo-0 4790m 3257Mi ds-idrepo-1 4693m 3106Mi ds-idrepo-2 4640m 3038Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1214Mi lodemon-5d46798758-5h2st 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6872m 919Mi 00:57:45 DEBUG "metric": { 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG }, 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "pod": "ds-cts-0" 00:57:45 DEBUG }, 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "result": [ 00:57:45 DEBUG "value": [ 00:57:45 DEBUG { 00:57:45 DEBUG "metric": { 00:57:45 DEBUG --- stderr --- 00:57:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG "value": [ 00:57:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG "pod": "ds-cts-0" 00:57:45 DEBUG { 00:57:45 DEBUG 00:57:45 DEBUG "metric": { 00:57:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:57:45 DEBUG 00:57:45 DEBUG }, 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "metric": { 00:57:45 DEBUG 00:57:45 DEBUG "instance": "10.142.0.107:9100" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG }, 00:57:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:57:45 DEBUG 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG "value": [ 00:57:45 DEBUG }, 00:57:45 DEBUG [...] 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "instance": "10.142.0.43:9100" 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "pod": "ds-idrepo-2" 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:57:45 DEBUG [...] 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:57:45 DEBUG }, 00:57:45 DEBUG "pod": "ds-idrepo-2" 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG }, 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG }, 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG }, 00:57:45 DEBUG 00:57:45 DEBUG "pod": "ds-cts-2" 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG 00:57:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:57:45 DEBUG }, 00:57:45 DEBUG [...] 00:57:45 DEBUG "0" 00:57:45 DEBUG "pod": "ds-cts-2" 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG 00:57:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 00:57:45 DEBUG }, 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG "47310984.36666666" 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG "20.076" 00:57:45 DEBUG "instance": "10.142.0.43:9100" 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG "1921.6333333333332" 00:57:45 DEBUG ] 00:57:45 DEBUG "0" 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG 00:57:45 DEBUG }, 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG }, 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG "1744.7" 00:57:45 DEBUG 00:57:45 DEBUG "value": [ 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG "value": [ 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG "1063129088" 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG "1921.8333333333333" 00:57:45 DEBUG ] 00:57:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG } 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG "0" 00:57:45 DEBUG ] 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG }, 00:57:45 DEBUG "0.000057542222222222224" 00:57:45 DEBUG ] 00:57:45 DEBUG "1.140333333333333" 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG "value": [ 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG 1713484665, 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG "1609820" 00:57:45 DEBUG ] 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 DEBUG ] 00:57:45 DEBUG } 00:57:45 DEBUG } 00:57:45 INFO 00:57:45 INFO [loop_until]: kubectl --namespace=xlou top node 00:57:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:57:45 INFO [loop_until]: OK (rc = 0) 00:57:45 DEBUG --- stdout --- 00:57:45 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5921Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4659m 15% 6657Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8418m 28% 11601Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3209Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9579m 32% 14782Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 142m 0% 5203Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10321m 34% 12174Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28038m 93% 3134Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9786m 32% 11253Mi 10% gke-xlou-cdm-ds-0f601751-j488 5001m 16% 6084Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8822m 29% 16314Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5251m 17% 6726Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8504m 28% 14253Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7164m 45% 4670Mi 7% 00:57:45 DEBUG --- stderr --- 00:57:45 DEBUG 00:58:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484724" 00:58:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484724" 00:58:44 ERROR [http_cmd]: ERROR 00:58:44 INFO [http_cmd]: http status code OK 00:58:44 ERROR --- status code --- 00:58:44 DEBUG --- status code --- 00:58:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:58:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:44 ERROR --- http response --- 00:58:44 DEBUG --- http response --- 00:58:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:58:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:58:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:58:44 DEBUG ----- output ----- 00:58:44 DEBUG { 00:58:44 DEBUG "status": "success", 00:58:44 DEBUG "data": { 00:58:44 DEBUG "resultType": "vector", 00:58:44 DEBUG "result": [ 00:58:44 DEBUG { 00:58:44 DEBUG "metric": { 00:58:44 DEBUG "instance": "10.142.0.107:9100" 00:58:44 DEBUG }, 00:58:44 DEBUG "value": [ 00:58:44 DEBUG 00:58:44 DEBUG 00:58:44 DEBUG 00:58:44 DEBUG [...] 00:58:44 DEBUG 00:58:44 DEBUG 00:58:44 DEBUG 00:58:44 DEBUG "instance": "10.142.0.43:9100" 00:58:44 DEBUG }, 00:58:44 DEBUG "value": [ 00:58:44 DEBUG 1713484724, 00:58:44 DEBUG "0.9562000000000012" 00:58:44 DEBUG ] 00:58:44 DEBUG } 00:58:44 DEBUG ] 00:58:44 DEBUG } 00:58:44 DEBUG } 00:58:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484724" 00:58:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484724" 00:58:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484724" 00:58:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484724" 00:58:44 INFO [http_cmd]: http status code OK 00:58:44 DEBUG --- status code --- 00:58:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:44 DEBUG --- http response --- 00:58:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:58:44 INFO [http_cmd]: http status code OK 00:58:44 DEBUG --- status code --- 00:58:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:44 DEBUG --- http response --- 00:58:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:58:44 DEBUG ----- output ----- 00:58:44 DEBUG { 00:58:44 DEBUG "status": "success", 00:58:44 DEBUG "data": { 00:58:44 DEBUG "resultType": "vector", 00:58:44 DEBUG "result": [ 00:58:44 DEBUG { 00:58:44 DEBUG "metric": { 00:58:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:58:44 DEBUG }, 00:58:44 DEBUG "value": [ 00:58:44 DEBUG 00:58:44 DEBUG 00:58:44 DEBUG 00:58:44 DEBUG [...] 00:58:44 DEBUG 00:58:44 DEBUG 00:58:44 DEBUG 00:58:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:58:44 DEBUG }, 00:58:44 DEBUG "value": [ 00:58:44 DEBUG 1713484724, 00:58:44 DEBUG "499.84745866129725" 00:58:44 DEBUG ] 00:58:44 DEBUG } 00:58:44 DEBUG ] 00:58:44 DEBUG } 00:58:44 DEBUG } 00:58:44 INFO [http_cmd]: http status code OK 00:58:44 DEBUG --- status code --- 00:58:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:44 DEBUG --- http response --- 00:58:44 INFO [http_cmd]: http status code OK 00:58:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:44 DEBUG --- status code --- 00:58:44 DEBUG ----- output ----- 00:58:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:44 DEBUG { 00:58:44 DEBUG --- http response --- 00:58:44 DEBUG "status": "success", 00:58:44 DEBUG "data": { 00:58:44 DEBUG "resultType": "vector", 00:58:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:58:44 DEBUG "result": [ 00:58:44 DEBUG ----- output ----- 00:58:44 DEBUG { 00:58:44 DEBUG { 00:58:44 DEBUG "metric": { 00:58:44 DEBUG "status": "success", 00:58:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:58:44 DEBUG "data": { 00:58:44 DEBUG }, 00:58:44 DEBUG "resultType": "vector", 00:58:44 DEBUG "value": [ 00:58:44 DEBUG "result": [ 00:58:44 DEBUG 00:58:44 DEBUG { 00:58:44 DEBUG 00:58:44 DEBUG "metric": { 00:58:44 DEBUG 00:58:44 DEBUG "instance": "10.142.0.107:9100" 00:58:44 DEBUG [...] 00:58:44 DEBUG }, 00:58:44 DEBUG 00:58:44 DEBUG "value": [ 00:58:44 DEBUG 00:58:44 DEBUG 00:58:44 DEBUG 00:58:44 DEBUG 00:58:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:58:44 DEBUG 00:58:44 DEBUG }, 00:58:44 DEBUG [...] 00:58:44 DEBUG "value": [ 00:58:44 DEBUG 00:58:44 DEBUG 1713484724, 00:58:44 DEBUG 00:58:44 DEBUG "0" 00:58:44 DEBUG 00:58:44 DEBUG ] 00:58:44 DEBUG "instance": "10.142.0.43:9100" 00:58:44 DEBUG } 00:58:44 DEBUG }, 00:58:44 DEBUG ] 00:58:44 DEBUG "value": [ 00:58:44 DEBUG } 00:58:44 DEBUG } 00:58:44 DEBUG 1713484724, 00:58:44 DEBUG "307557116928" 00:58:44 DEBUG ] 00:58:44 DEBUG } 00:58:44 DEBUG ] 00:58:44 DEBUG } 00:58:44 DEBUG } 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG { 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "data": { 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "result": [ 00:58:45 DEBUG { 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "pod": "ds-idrepo-0" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "pod": "ds-cts-2" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG "0" 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484725" 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG { 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "data": { 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "result": [ 00:58:45 DEBUG { 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "instance": "10.142.0.107:9100" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "instance": "10.142.0.43:9100" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG "47263213.733333334" 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG { 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "data": { 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "result": [ 00:58:45 DEBUG { 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG "28741.228873024593" 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG { 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "data": { 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "result": [ 00:58:45 DEBUG { 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "pod": "ds-cts-0" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "pod": "ds-cts-2" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG "0.8" 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG { 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "data": { 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "result": [ 00:58:45 DEBUG { 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "pod": "ds-idrepo-0" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "pod": "ds-idrepo-2" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG "0" 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG { 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "data": { 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "result": [ 00:58:45 DEBUG { 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG "1926.9333333333334" 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG { 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "data": { 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "result": [ 00:58:45 DEBUG { 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG "9.274270425285243" 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484725" 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG { 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "data": { 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "result": [ 00:58:45 DEBUG { 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG "6.9267920239134115" 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG } 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG } 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG { 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "data": { 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "result": [ 00:58:45 DEBUG { 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG "0" 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 INFO 00:58:45 INFO [loop_until]: kubectl --namespace=xlou top pods 00:58:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484725" 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484725" 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484725" 00:58:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484725" 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG --- status code --- 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG --- status code --- 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG --- status code --- 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG --- status code --- 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG { 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:45 DEBUG { 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG { 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:58:45 DEBUG { 00:58:45 INFO [http_cmd]: http status code OK 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG { 00:58:45 DEBUG { 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "data": { 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG { 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG --- status code --- 00:58:45 DEBUG "data": { 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG { 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:58:45 DEBUG { 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "data": { 00:58:45 DEBUG { 00:58:45 DEBUG "data": { 00:58:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "data": { 00:58:45 DEBUG "data": { 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "result": [ 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "data": { 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG --- http response --- 00:58:45 DEBUG "result": [ 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "data": { 00:58:45 DEBUG { 00:58:45 DEBUG { 00:58:45 DEBUG "data": { 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "result": [ 00:58:45 DEBUG "data": { 00:58:45 DEBUG "result": [ 00:58:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:58:45 DEBUG { 00:58:45 DEBUG "result": [ 00:58:45 DEBUG "result": [ 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG "result": [ 00:58:45 DEBUG { 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG { 00:58:45 DEBUG ----- output ----- 00:58:45 DEBUG "metric": { 00:58:45 DEBUG { 00:58:45 DEBUG { 00:58:45 DEBUG "result": [ 00:58:45 DEBUG "instance": "10.142.0.107:9100" 00:58:45 DEBUG "data": { 00:58:45 DEBUG "result": [ 00:58:45 DEBUG { 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "result": [ 00:58:45 DEBUG "metric": { 00:58:45 DEBUG { 00:58:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "metric": { 00:58:45 DEBUG { 00:58:45 DEBUG }, 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG { 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "instance": "10.142.0.107:9100" 00:58:45 DEBUG { 00:58:45 DEBUG "pod": "ds-idrepo-0" 00:58:45 DEBUG "status": "success", 00:58:45 DEBUG }, 00:58:45 DEBUG "pod": "ds-cts-0" 00:58:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "value": [ 00:58:45 DEBUG "result": [ 00:58:45 DEBUG "metric": { 00:58:45 DEBUG "pod": "ds-idrepo-0" 00:58:45 DEBUG }, 00:58:45 DEBUG "metric": { 00:58:45 DEBUG }, 00:58:45 DEBUG "data": { 00:58:45 DEBUG "value": [ 00:58:45 DEBUG }, 00:58:45 DEBUG }, 00:58:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:58:45 DEBUG 00:58:45 DEBUG { 00:58:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG "pod": "ds-cts-0" 00:58:45 DEBUG "value": [ 00:58:45 DEBUG "resultType": "vector", 00:58:45 DEBUG 00:58:45 DEBUG "value": [ 00:58:45 DEBUG "value": [ 00:58:45 DEBUG }, 00:58:45 DEBUG 00:58:45 DEBUG "metric": { 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG }, 00:58:45 DEBUG 00:58:45 DEBUG "result": [ 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG { 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG }, 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "metric": { 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG }, 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "instance": "10.142.0.43:9100" 00:58:45 DEBUG }, 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 INFO [loop_until]: OK (rc = 0) 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG 00:58:45 DEBUG --- stdout --- 00:58:45 DEBUG "instance": "10.142.0.43:9100" 00:58:45 DEBUG 00:58:45 DEBUG "pod": "ds-idrepo-2" 00:58:45 DEBUG 00:58:45 DEBUG }, 00:58:45 DEBUG "pod": "ds-cts-2" 00:58:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:58:45 DEBUG 00:58:45 DEBUG [...] 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG 00:58:45 DEBUG "pod": "ds-idrepo-2" 00:58:45 DEBUG }, 00:58:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10144m 9382Mi am-6fd554f8f7-hvvkw 8367m 8873Mi am-6fd554f8f7-pmj4b 9101m 9028Mi ds-cts-0 9431m 12269Mi ds-cts-1 7926m 12236Mi ds-cts-2 8151m 12295Mi ds-idrepo-0 4774m 3258Mi ds-idrepo-1 4692m 3105Mi ds-idrepo-2 4640m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1214Mi lodemon-5d46798758-5h2st 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6859m 921Mi 00:58:45 DEBUG 00:58:45 DEBUG }, 00:58:45 DEBUG 00:58:45 DEBUG "value": [ 00:58:45 DEBUG }, 00:58:45 DEBUG }, 00:58:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:58:45 DEBUG 00:58:45 DEBUG "47268721.53333333" 00:58:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:58:45 DEBUG }, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG --- stderr --- 00:58:45 DEBUG "pod": "ds-cts-2" 00:58:45 DEBUG "value": [ 00:58:45 DEBUG [...] 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG "value": [ 00:58:45 DEBUG }, 00:58:45 DEBUG 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG }, 00:58:45 DEBUG 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG 00:58:45 DEBUG "1927" 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG "value": [ 00:58:45 DEBUG 00:58:45 DEBUG }, 00:58:45 DEBUG ] 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG "1.072" 00:58:45 DEBUG "value": [ 00:58:45 DEBUG "0" 00:58:45 DEBUG 00:58:45 DEBUG ] 00:58:45 DEBUG "0" 00:58:45 DEBUG "20.076" 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:58:45 DEBUG "value": [ 00:58:45 DEBUG } 00:58:45 DEBUG "0" 00:58:45 DEBUG ] 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG ] 00:58:45 DEBUG 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG ] 00:58:45 DEBUG "1725527.5" 00:58:45 DEBUG }, 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG "446.26666666666665" 00:58:45 DEBUG } 00:58:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG "value": [ 00:58:45 DEBUG "0" 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG ] 00:58:45 DEBUG ] 00:58:45 DEBUG }, 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG ] 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG "value": [ 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG "1927.5" 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG 1713484725, 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG "1058598912" 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG ] 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 DEBUG } 00:58:45 INFO 00:58:45 INFO [loop_until]: kubectl --namespace=xlou top node 00:58:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:58:46 INFO [loop_until]: OK (rc = 0) 00:58:46 DEBUG --- stdout --- 00:58:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5929Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4636m 15% 6653Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8465m 28% 11600Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3224Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9350m 31% 14697Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5204Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10377m 34% 12175Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28007m 93% 3144Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10041m 33% 11258Mi 10% gke-xlou-cdm-ds-0f601751-j488 5017m 16% 6085Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8298m 27% 16242Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5270m 17% 6736Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8203m 27% 14195Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7066m 44% 4669Mi 7% 00:58:46 DEBUG --- stderr --- 00:58:46 DEBUG 00:59:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484784" 00:59:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484784" 00:59:44 INFO [http_cmd]: http status code OK 00:59:44 DEBUG --- status code --- 00:59:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:44 DEBUG --- http response --- 00:59:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:59:44 DEBUG ----- output ----- 00:59:44 DEBUG { 00:59:44 DEBUG "status": "success", 00:59:44 DEBUG "data": { 00:59:44 DEBUG "resultType": "vector", 00:59:44 DEBUG "result": [ 00:59:44 DEBUG { 00:59:44 DEBUG "metric": { 00:59:44 DEBUG "instance": "10.142.0.107:9100" 00:59:44 DEBUG }, 00:59:44 DEBUG "value": [ 00:59:44 DEBUG 00:59:44 DEBUG 00:59:44 DEBUG 00:59:44 DEBUG [...] 00:59:44 DEBUG 00:59:44 DEBUG 00:59:44 DEBUG 00:59:44 DEBUG "instance": "10.142.0.43:9100" 00:59:44 DEBUG }, 00:59:44 DEBUG "value": [ 00:59:44 DEBUG 1713484784, 00:59:44 DEBUG "0.9522333333333335" 00:59:44 DEBUG ] 00:59:44 DEBUG } 00:59:44 DEBUG ] 00:59:44 DEBUG } 00:59:44 DEBUG } 00:59:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484784" 00:59:44 ERROR [http_cmd]: ERROR 00:59:44 ERROR --- status code --- 00:59:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 00:59:44 ERROR --- http response --- 00:59:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 00:59:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 00:59:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484784" 00:59:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484784" 00:59:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484784" 00:59:44 INFO [http_cmd]: http status code OK 00:59:44 DEBUG --- status code --- 00:59:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:44 DEBUG --- http response --- 00:59:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 00:59:44 INFO [http_cmd]: http status code OK 00:59:44 DEBUG --- status code --- 00:59:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:44 DEBUG --- http response --- 00:59:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:59:44 DEBUG ----- output ----- 00:59:44 DEBUG { 00:59:44 DEBUG "status": "success", 00:59:44 DEBUG "data": { 00:59:44 DEBUG "resultType": "vector", 00:59:44 DEBUG "result": [ 00:59:44 DEBUG { 00:59:44 DEBUG "metric": { 00:59:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:59:44 DEBUG }, 00:59:44 DEBUG "value": [ 00:59:44 DEBUG 00:59:44 DEBUG 00:59:44 DEBUG 00:59:44 DEBUG [...] 00:59:44 DEBUG 00:59:44 DEBUG 00:59:44 DEBUG 00:59:44 DEBUG "pod": "ds-cts-2" 00:59:44 DEBUG }, 00:59:44 DEBUG "value": [ 00:59:44 DEBUG 1713484784, 00:59:44 DEBUG "20569608.746980853" 00:59:44 DEBUG ] 00:59:44 DEBUG } 00:59:44 DEBUG ] 00:59:44 DEBUG } 00:59:44 DEBUG } 00:59:44 INFO [http_cmd]: http status code OK 00:59:44 DEBUG --- status code --- 00:59:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:44 INFO [http_cmd]: http status code OK 00:59:44 DEBUG --- http response --- 00:59:44 DEBUG --- status code --- 00:59:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:44 DEBUG ----- output ----- 00:59:44 DEBUG --- http response --- 00:59:44 DEBUG { 00:59:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:59:44 DEBUG "status": "success", 00:59:44 DEBUG ----- output ----- 00:59:44 DEBUG "data": { 00:59:44 DEBUG { 00:59:44 DEBUG "resultType": "vector", 00:59:44 DEBUG "result": [ 00:59:44 DEBUG { 00:59:44 DEBUG "status": "success", 00:59:44 DEBUG "metric": { 00:59:44 DEBUG "data": { 00:59:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:59:44 DEBUG "resultType": "vector", 00:59:44 DEBUG }, 00:59:44 DEBUG "value": [ 00:59:44 DEBUG "result": [ 00:59:44 DEBUG 00:59:44 DEBUG { 00:59:44 DEBUG 00:59:44 DEBUG "metric": { 00:59:44 DEBUG 00:59:44 DEBUG "instance": "10.142.0.107:9100" 00:59:44 DEBUG [...] 00:59:44 DEBUG }, 00:59:44 DEBUG 00:59:44 DEBUG "value": [ 00:59:44 DEBUG 00:59:44 DEBUG 00:59:44 DEBUG 00:59:44 DEBUG 00:59:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:59:44 DEBUG 00:59:44 DEBUG }, 00:59:44 DEBUG [...] 00:59:44 DEBUG "value": [ 00:59:44 DEBUG 00:59:44 DEBUG 1713484784, 00:59:44 DEBUG 00:59:44 DEBUG "0" 00:59:44 DEBUG 00:59:44 DEBUG ] 00:59:44 DEBUG "instance": "10.142.0.43:9100" 00:59:44 DEBUG } 00:59:44 DEBUG }, 00:59:44 DEBUG ] 00:59:44 DEBUG "value": [ 00:59:44 DEBUG } 00:59:44 DEBUG 1713484784, 00:59:44 DEBUG } 00:59:44 DEBUG "309934918656" 00:59:44 DEBUG ] 00:59:44 DEBUG } 00:59:44 DEBUG ] 00:59:44 DEBUG } 00:59:44 DEBUG } 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "result": [ 00:59:45 DEBUG { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "pod": "ds-idrepo-0" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "pod": "ds-cts-2" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG "0" 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484785" 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "result": [ 00:59:45 DEBUG { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "instance": "10.142.0.107:9100" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "instance": "10.142.0.43:9100" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG "47913474" 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "result": [ 00:59:45 DEBUG { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "pod": "ds-cts-0" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "pod": "ds-cts-2" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG "0.8333333333333334" 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "result": [ 00:59:45 DEBUG { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG "0" 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "result": [ 00:59:45 DEBUG { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "pod": "ds-idrepo-0" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "pod": "ds-idrepo-2" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG "0" 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "result": [ 00:59:45 DEBUG { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG "1918.8" 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "result": [ 00:59:45 DEBUG { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "pod": "idm-b85f9d978-wnhxb" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG "0" 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484785" 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "result": [ 00:59:45 DEBUG { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG "0" 00:59:45 DEBUG ] 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG } 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG ] 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG } 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG } 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "result": [ 00:59:45 DEBUG { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG "6.934714825884303" 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG { 00:59:45 DEBUG { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "data": { 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "result": [ 00:59:45 DEBUG "result": [ 00:59:45 DEBUG { 00:59:45 DEBUG { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "pod": "ds-idrepo-0" 00:59:45 DEBUG "pod": "ds-idrepo-0" 00:59:45 DEBUG }, 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "pod": "ds-idrepo-2" 00:59:45 DEBUG "pod": "ds-idrepo-2" 00:59:45 DEBUG }, 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG "0" 00:59:45 DEBUG "0" 00:59:45 DEBUG ] 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "result": [ 00:59:45 DEBUG { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "pod": "ds-cts-2" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG "10283673.648336543" 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 INFO 00:59:45 INFO [loop_until]: kubectl --namespace=xlou top pods 00:59:45 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484785" 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484785" 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484785" 00:59:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484785" 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "result": [ 00:59:45 DEBUG { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:59:45 DEBUG }, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG }, 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG "value": [ 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG "1919.0333333333333" 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG ] 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:59:45 INFO [http_cmd]: http status code OK 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG --- status code --- 00:59:45 DEBUG } 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG { 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG { 00:59:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 00:59:45 DEBUG } 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG { 00:59:45 DEBUG { 00:59:45 DEBUG { 00:59:45 DEBUG { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG --- http response --- 00:59:45 DEBUG { 00:59:45 DEBUG "data": { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "data": { 00:59:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "data": { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG ----- output ----- 00:59:45 DEBUG "data": { 00:59:45 DEBUG "result": [ 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "data": { 00:59:45 DEBUG "result": [ 00:59:45 DEBUG "result": [ 00:59:45 DEBUG { 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG { 00:59:45 DEBUG "result": [ 00:59:45 DEBUG "result": [ 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG { 00:59:45 DEBUG { 00:59:45 DEBUG "status": "success", 00:59:45 DEBUG "result": [ 00:59:45 DEBUG "metric": { 00:59:45 DEBUG { 00:59:45 DEBUG { 00:59:45 DEBUG "result": [ 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "data": { 00:59:45 DEBUG { 00:59:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "metric": { 00:59:45 DEBUG { 00:59:45 DEBUG "pod": "ds-cts-0" 00:59:45 DEBUG "instance": "10.142.0.107:9100" 00:59:45 DEBUG "resultType": "vector", 00:59:45 DEBUG "metric": { 00:59:45 DEBUG }, 00:59:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:59:45 DEBUG "instance": "10.142.0.107:9100" 00:59:45 DEBUG "metric": { 00:59:45 DEBUG }, 00:59:45 DEBUG }, 00:59:45 DEBUG "result": [ 00:59:45 DEBUG "pod": "ds-cts-0" 00:59:45 DEBUG "value": [ 00:59:45 DEBUG }, 00:59:45 DEBUG }, 00:59:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 00:59:45 DEBUG "value": [ 00:59:45 DEBUG "value": [ 00:59:45 DEBUG { 00:59:45 DEBUG }, 00:59:45 DEBUG 00:59:45 DEBUG "value": [ 00:59:45 DEBUG "value": [ 00:59:45 DEBUG }, 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "metric": { 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG }, 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG [...] 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG 00:59:45 DEBUG "pod": "ds-cts-2" 00:59:45 DEBUG "instance": "10.142.0.43:9100" 00:59:45 DEBUG [...] 00:59:45 DEBUG 00:59:45 DEBUG }, 00:59:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:59:45 DEBUG "instance": "10.142.0.43:9100" 00:59:45 DEBUG 00:59:45 DEBUG }, 00:59:45 DEBUG }, 00:59:45 DEBUG 00:59:45 DEBUG "pod": "ds-cts-2" 00:59:45 DEBUG "value": [ 00:59:45 DEBUG }, 00:59:45 DEBUG }, 00:59:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 00:59:45 DEBUG "value": [ 00:59:45 DEBUG "value": [ 00:59:45 DEBUG 00:59:45 DEBUG }, 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG "value": [ 00:59:45 DEBUG }, 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG 00:59:45 DEBUG "value": [ 00:59:45 DEBUG "20.076" 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG "value": [ 00:59:45 DEBUG "2223.5333333333333" 00:59:45 DEBUG "47918835.2" 00:59:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG ] 00:59:45 DEBUG "1918.8666666666666" 00:59:45 DEBUG "1.036" 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG ] 00:59:45 DEBUG ] 00:59:45 DEBUG }, 00:59:45 DEBUG "0.000028771111111111112" 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG ] 00:59:45 DEBUG "1840477" 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG "value": [ 00:59:45 DEBUG ] 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG ] 00:59:45 DEBUG ] 00:59:45 DEBUG 1713484785, 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG "1061380096" 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG ] 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 DEBUG } 00:59:45 INFO [loop_until]: OK (rc = 0) 00:59:45 DEBUG --- stdout --- 00:59:45 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10354m 9383Mi am-6fd554f8f7-hvvkw 8360m 8874Mi am-6fd554f8f7-pmj4b 9258m 9028Mi ds-cts-0 9262m 12410Mi ds-cts-1 8594m 12269Mi ds-cts-2 9453m 12366Mi ds-idrepo-0 4786m 3257Mi ds-idrepo-1 4739m 3106Mi ds-idrepo-2 4465m 3038Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6891m 921Mi 00:59:45 DEBUG --- stderr --- 00:59:45 DEBUG 00:59:46 INFO 00:59:46 INFO [loop_until]: kubectl --namespace=xlou top node 00:59:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 00:59:46 INFO [loop_until]: OK (rc = 0) 00:59:46 DEBUG --- stdout --- 00:59:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5928Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4612m 15% 6648Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8417m 28% 11611Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3232Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9844m 32% 14882Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 5223Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10480m 35% 12172Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28083m 94% 3127Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9816m 32% 11250Mi 10% gke-xlou-cdm-ds-0f601751-j488 5003m 16% 6085Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8258m 27% 16270Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5296m 17% 6742Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9082m 30% 14298Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7075m 44% 4669Mi 7% 00:59:46 DEBUG --- stderr --- 00:59:46 DEBUG 01:00:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484844" 01:00:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484844" 01:00:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484844" 01:00:44 INFO [http_cmd]: http status code OK 01:00:44 DEBUG --- status code --- 01:00:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:44 DEBUG --- http response --- 01:00:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:00:44 ERROR [http_cmd]: ERROR 01:00:44 DEBUG ----- output ----- 01:00:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484844" 01:00:44 ERROR --- status code --- 01:00:44 DEBUG { 01:00:44 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:00:44 DEBUG "status": "success", 01:00:44 ERROR --- http response --- 01:00:44 DEBUG "data": { 01:00:44 DEBUG "resultType": "vector", 01:00:44 DEBUG "result": [ 01:00:44 DEBUG { 01:00:44 DEBUG "metric": { 01:00:44 DEBUG "instance": "10.142.0.107:9100" 01:00:44 DEBUG }, 01:00:44 DEBUG "value": [ 01:00:44 DEBUG 01:00:44 DEBUG 01:00:44 DEBUG 01:00:44 DEBUG [...] 01:00:44 DEBUG 01:00:44 DEBUG 01:00:44 DEBUG 01:00:44 DEBUG "instance": "10.142.0.43:9100" 01:00:44 DEBUG }, 01:00:44 DEBUG "value": [ 01:00:44 DEBUG 1713484844, 01:00:44 DEBUG "0.9632333333333358" 01:00:44 DEBUG ] 01:00:44 DEBUG } 01:00:44 DEBUG ] 01:00:44 DEBUG } 01:00:44 DEBUG } 01:00:44 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:00:44 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:00:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484844" 01:00:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484844" 01:00:44 INFO [http_cmd]: http status code OK 01:00:44 DEBUG --- status code --- 01:00:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:44 DEBUG --- http response --- 01:00:44 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:00:44 INFO [http_cmd]: http status code OK 01:00:44 DEBUG --- status code --- 01:00:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:44 DEBUG --- http response --- 01:00:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:00:44 DEBUG ----- output ----- 01:00:44 DEBUG { 01:00:44 DEBUG "status": "success", 01:00:44 DEBUG "data": { 01:00:44 DEBUG "resultType": "vector", 01:00:44 DEBUG "result": [ 01:00:44 DEBUG { 01:00:44 DEBUG "metric": { 01:00:44 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:00:44 DEBUG }, 01:00:44 DEBUG "value": [ 01:00:44 DEBUG 01:00:44 DEBUG 01:00:44 DEBUG 01:00:44 DEBUG [...] 01:00:44 DEBUG 01:00:44 DEBUG 01:00:44 DEBUG 01:00:44 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:00:44 DEBUG }, 01:00:44 DEBUG "value": [ 01:00:44 DEBUG 1713484844, 01:00:44 DEBUG "0" 01:00:44 DEBUG ] 01:00:44 DEBUG } 01:00:44 DEBUG ] 01:00:44 DEBUG } 01:00:44 DEBUG } 01:00:44 INFO [http_cmd]: http status code OK 01:00:44 INFO [http_cmd]: http status code OK 01:00:44 DEBUG --- status code --- 01:00:44 DEBUG --- status code --- 01:00:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:44 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:44 DEBUG --- http response --- 01:00:44 DEBUG --- http response --- 01:00:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:00:44 DEBUG ----- output ----- 01:00:44 DEBUG { 01:00:44 DEBUG "status": "success", 01:00:44 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:44 DEBUG "data": { 01:00:44 DEBUG ----- output ----- 01:00:44 DEBUG "resultType": "vector", 01:00:44 DEBUG { 01:00:44 DEBUG "result": [ 01:00:44 DEBUG "status": "success", 01:00:44 DEBUG { 01:00:44 DEBUG "data": { 01:00:44 DEBUG "metric": { 01:00:44 DEBUG "resultType": "vector", 01:00:44 DEBUG "instance": "10.142.0.107:9100" 01:00:44 DEBUG "result": [ 01:00:44 DEBUG }, 01:00:44 DEBUG { 01:00:44 DEBUG "value": [ 01:00:44 DEBUG "metric": { 01:00:44 DEBUG 01:00:44 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:00:44 DEBUG 01:00:44 DEBUG }, 01:00:44 DEBUG 01:00:44 DEBUG "value": [ 01:00:44 DEBUG [...] 01:00:44 DEBUG 01:00:44 DEBUG 01:00:44 DEBUG 01:00:44 DEBUG 01:00:44 DEBUG 01:00:44 DEBUG 01:00:44 DEBUG [...] 01:00:44 DEBUG "instance": "10.142.0.43:9100" 01:00:44 DEBUG 01:00:44 DEBUG }, 01:00:44 DEBUG 01:00:44 DEBUG "value": [ 01:00:44 DEBUG 01:00:44 DEBUG 1713484844, 01:00:44 DEBUG "312332602368" 01:00:44 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:00:44 DEBUG ] 01:00:44 DEBUG }, 01:00:44 DEBUG } 01:00:44 DEBUG "value": [ 01:00:44 DEBUG ] 01:00:44 DEBUG 1713484844, 01:00:44 DEBUG } 01:00:44 DEBUG "0" 01:00:44 DEBUG } 01:00:44 DEBUG ] 01:00:44 DEBUG } 01:00:44 DEBUG ] 01:00:44 DEBUG } 01:00:44 DEBUG } 01:00:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484845" 01:00:45 INFO [http_cmd]: http status code OK 01:00:45 DEBUG --- status code --- 01:00:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:45 DEBUG --- http response --- 01:00:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:00:45 DEBUG ----- output ----- 01:00:45 DEBUG { 01:00:45 DEBUG "status": "success", 01:00:45 DEBUG "data": { 01:00:45 DEBUG "resultType": "vector", 01:00:45 DEBUG "result": [ 01:00:45 DEBUG { 01:00:45 DEBUG "metric": { 01:00:45 DEBUG "pod": "ds-idrepo-0" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG [...] 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG "pod": "ds-cts-2" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 1713484845, 01:00:45 DEBUG "0" 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG } 01:00:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484845" 01:00:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484845" 01:00:45 INFO [http_cmd]: http status code OK 01:00:45 DEBUG --- status code --- 01:00:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:45 DEBUG --- http response --- 01:00:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:00:45 DEBUG ----- output ----- 01:00:45 DEBUG { 01:00:45 DEBUG "status": "success", 01:00:45 DEBUG "data": { 01:00:45 DEBUG "resultType": "vector", 01:00:45 DEBUG "result": [ 01:00:45 DEBUG { 01:00:45 DEBUG "metric": { 01:00:45 DEBUG "instance": "10.142.0.107:9100" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG [...] 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG "instance": "10.142.0.43:9100" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 1713484845, 01:00:45 DEBUG "47359139.63333333" 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG } 01:00:45 INFO [http_cmd]: http status code OK 01:00:45 DEBUG --- status code --- 01:00:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:45 DEBUG --- http response --- 01:00:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:00:45 DEBUG ----- output ----- 01:00:45 DEBUG { 01:00:45 DEBUG "status": "success", 01:00:45 DEBUG "data": { 01:00:45 DEBUG "resultType": "vector", 01:00:45 DEBUG "result": [ 01:00:45 DEBUG { 01:00:45 DEBUG "metric": { 01:00:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG [...] 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 1713484845, 01:00:45 DEBUG "36059.69265191659" 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG } 01:00:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484845" 01:00:45 INFO [http_cmd]: http status code OK 01:00:45 DEBUG --- status code --- 01:00:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:45 DEBUG --- http response --- 01:00:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:00:45 DEBUG ----- output ----- 01:00:45 DEBUG { 01:00:45 DEBUG "status": "success", 01:00:45 DEBUG "data": { 01:00:45 DEBUG "resultType": "vector", 01:00:45 DEBUG "result": [ 01:00:45 DEBUG { 01:00:45 DEBUG "metric": { 01:00:45 DEBUG "pod": "ds-cts-0" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG [...] 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG "pod": "ds-cts-2" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 1713484845, 01:00:45 DEBUG "1.1666666666666667" 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG } 01:00:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484845" 01:00:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484845" 01:00:45 INFO [http_cmd]: http status code OK 01:00:45 DEBUG --- status code --- 01:00:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:45 DEBUG --- http response --- 01:00:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:45 DEBUG ----- output ----- 01:00:45 DEBUG { 01:00:45 DEBUG "status": "success", 01:00:45 DEBUG "data": { 01:00:45 DEBUG "resultType": "vector", 01:00:45 DEBUG "result": [ 01:00:45 DEBUG { 01:00:45 DEBUG "metric": { 01:00:45 DEBUG "pod": "ds-idrepo-0" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG [...] 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG "pod": "ds-idrepo-2" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 1713484845, 01:00:45 DEBUG "0" 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG } 01:00:45 INFO [http_cmd]: http status code OK 01:00:45 DEBUG --- status code --- 01:00:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:45 DEBUG --- http response --- 01:00:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:45 DEBUG ----- output ----- 01:00:45 DEBUG { 01:00:45 DEBUG "status": "success", 01:00:45 DEBUG "data": { 01:00:45 DEBUG "resultType": "vector", 01:00:45 DEBUG "result": [ 01:00:45 DEBUG { 01:00:45 DEBUG "metric": { 01:00:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG [...] 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 1713484845, 01:00:45 DEBUG "1919.4666666666667" 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG } 01:00:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484845" 01:00:45 INFO [http_cmd]: http status code OK 01:00:45 DEBUG --- status code --- 01:00:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:45 DEBUG --- http response --- 01:00:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:00:45 DEBUG ----- output ----- 01:00:45 DEBUG { 01:00:45 DEBUG "status": "success", 01:00:45 DEBUG "data": { 01:00:45 DEBUG "resultType": "vector", 01:00:45 DEBUG "result": [ 01:00:45 DEBUG { 01:00:45 DEBUG "metric": { 01:00:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG [...] 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 1713484845, 01:00:45 DEBUG "13.769992597044169" 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG } 01:00:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484845" 01:00:45 INFO [http_cmd]: http status code OK 01:00:45 DEBUG --- status code --- 01:00:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:45 DEBUG --- http response --- 01:00:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:00:45 DEBUG ----- output ----- 01:00:45 DEBUG { 01:00:45 DEBUG "status": "success", 01:00:45 DEBUG "data": { 01:00:45 DEBUG "resultType": "vector", 01:00:45 DEBUG "result": [ 01:00:45 DEBUG { 01:00:45 DEBUG "metric": { 01:00:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG [...] 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 1713484845, 01:00:45 DEBUG "6.940841307285013" 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG } 01:00:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484845" 01:00:45 INFO [http_cmd]: http status code OK 01:00:45 DEBUG --- status code --- 01:00:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:45 DEBUG --- http response --- 01:00:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:45 DEBUG ----- output ----- 01:00:45 DEBUG { 01:00:45 DEBUG "status": "success", 01:00:45 DEBUG "data": { 01:00:45 DEBUG "resultType": "vector", 01:00:45 DEBUG "result": [ 01:00:45 DEBUG { 01:00:45 DEBUG "metric": { 01:00:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG [...] 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 1713484845, 01:00:45 DEBUG "0" 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG } 01:00:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484845" 01:00:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484845" 01:00:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484845" 01:00:45 INFO [http_cmd]: http status code OK 01:00:45 INFO [http_cmd]: http status code OK 01:00:45 DEBUG --- status code --- 01:00:45 DEBUG --- status code --- 01:00:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:45 DEBUG --- http response --- 01:00:45 DEBUG --- http response --- 01:00:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:45 DEBUG ----- output ----- 01:00:45 DEBUG ----- output ----- 01:00:45 DEBUG { 01:00:45 DEBUG { 01:00:45 DEBUG "status": "success", 01:00:45 DEBUG "status": "success", 01:00:45 DEBUG "data": { 01:00:45 DEBUG "data": { 01:00:45 DEBUG "resultType": "vector", 01:00:45 DEBUG "resultType": "vector", 01:00:45 DEBUG "result": [ 01:00:45 DEBUG "result": [ 01:00:45 DEBUG { 01:00:45 DEBUG { 01:00:45 DEBUG "metric": { 01:00:45 DEBUG "metric": { 01:00:45 DEBUG "pod": "ds-idrepo-0" 01:00:45 DEBUG "pod": "ds-idrepo-0" 01:00:45 DEBUG }, 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 INFO [http_cmd]: http status code OK 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG --- status code --- 01:00:45 DEBUG [...] 01:00:45 DEBUG [...] 01:00:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG --- http response --- 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:00:45 DEBUG 01:00:45 DEBUG "pod": "ds-idrepo-2" 01:00:45 DEBUG ----- output ----- 01:00:45 DEBUG "pod": "ds-idrepo-2" 01:00:45 DEBUG }, 01:00:45 DEBUG { 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG "status": "success", 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 1713484845, 01:00:45 DEBUG "data": { 01:00:45 DEBUG 1713484845, 01:00:45 DEBUG "0" 01:00:45 DEBUG "resultType": "vector", 01:00:45 DEBUG "0" 01:00:45 DEBUG ] 01:00:45 DEBUG "result": [ 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG { 01:00:45 DEBUG } 01:00:45 DEBUG ] 01:00:45 DEBUG "metric": { 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:00:45 DEBUG } 01:00:45 DEBUG } 01:00:45 DEBUG }, 01:00:45 DEBUG } 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG [...] 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG 01:00:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:00:45 DEBUG }, 01:00:45 DEBUG "value": [ 01:00:45 DEBUG 1713484845, 01:00:45 DEBUG "0" 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG ] 01:00:45 DEBUG } 01:00:45 DEBUG } 01:00:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484846" 01:00:46 INFO [http_cmd]: http status code OK 01:00:46 DEBUG --- status code --- 01:00:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:46 DEBUG --- http response --- 01:00:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:46 DEBUG ----- output ----- 01:00:46 DEBUG { 01:00:46 DEBUG "status": "success", 01:00:46 DEBUG "data": { 01:00:46 DEBUG "resultType": "vector", 01:00:46 DEBUG "result": [ 01:00:46 DEBUG { 01:00:46 DEBUG "metric": { 01:00:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:00:46 DEBUG }, 01:00:46 DEBUG "value": [ 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG [...] 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:00:46 DEBUG }, 01:00:46 DEBUG "value": [ 01:00:46 DEBUG 1713484846, 01:00:46 DEBUG "1919.3" 01:00:46 DEBUG ] 01:00:46 DEBUG } 01:00:46 DEBUG ] 01:00:46 DEBUG } 01:00:46 DEBUG } 01:00:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484846" 01:00:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484846" 01:00:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484846" 01:00:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484846" 01:00:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484846" 01:00:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484846" 01:00:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484846" 01:00:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484846" 01:00:46 INFO [http_cmd]: http status code OK 01:00:46 DEBUG --- status code --- 01:00:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:46 DEBUG --- http response --- 01:00:46 INFO [http_cmd]: http status code OK 01:00:46 DEBUG --- status code --- 01:00:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:46 DEBUG --- http response --- 01:00:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:46 DEBUG ----- output ----- 01:00:46 DEBUG { 01:00:46 DEBUG "status": "success", 01:00:46 INFO [http_cmd]: http status code OK 01:00:46 INFO 01:00:46 DEBUG "data": { 01:00:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:46 INFO [http_cmd]: http status code OK 01:00:46 DEBUG --- status code --- 01:00:46 INFO [http_cmd]: http status code OK 01:00:46 INFO [http_cmd]: http status code OK 01:00:46 INFO [loop_until]: kubectl --namespace=xlou top pods 01:00:46 DEBUG "resultType": "vector", 01:00:46 DEBUG ----- output ----- 01:00:46 DEBUG --- status code --- 01:00:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:46 DEBUG --- status code --- 01:00:46 DEBUG --- status code --- 01:00:46 INFO [http_cmd]: http status code OK 01:00:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:00:46 DEBUG "result": [ 01:00:46 DEBUG { 01:00:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:46 DEBUG --- http response --- 01:00:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:46 DEBUG --- status code --- 01:00:46 DEBUG { 01:00:46 DEBUG "status": "success", 01:00:46 DEBUG --- http response --- 01:00:46 INFO [http_cmd]: http status code OK 01:00:46 DEBUG --- http response --- 01:00:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:00:46 DEBUG --- http response --- 01:00:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:46 DEBUG "metric": { 01:00:46 DEBUG "data": { 01:00:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:00:46 DEBUG --- status code --- 01:00:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:00:46 DEBUG ----- output ----- 01:00:46 DEBUG --- http response --- 01:00:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:00:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:00:46 DEBUG "resultType": "vector", 01:00:46 DEBUG ----- output ----- 01:00:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:00:46 DEBUG ----- output ----- 01:00:46 DEBUG { 01:00:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:00:46 DEBUG ----- output ----- 01:00:46 DEBUG }, 01:00:46 DEBUG "result": [ 01:00:46 DEBUG { 01:00:46 DEBUG --- http response --- 01:00:46 DEBUG { 01:00:46 DEBUG "status": "success", 01:00:46 DEBUG ----- output ----- 01:00:46 DEBUG { 01:00:46 DEBUG "value": [ 01:00:46 DEBUG { 01:00:46 DEBUG "status": "success", 01:00:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:00:46 DEBUG "status": "success", 01:00:46 DEBUG "data": { 01:00:46 DEBUG { 01:00:46 DEBUG "status": "success", 01:00:46 DEBUG 01:00:46 DEBUG "metric": { 01:00:46 DEBUG "data": { 01:00:46 DEBUG ----- output ----- 01:00:46 DEBUG "data": { 01:00:46 DEBUG "resultType": "vector", 01:00:46 DEBUG "status": "success", 01:00:46 DEBUG "data": { 01:00:46 DEBUG 01:00:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:00:46 DEBUG "resultType": "vector", 01:00:46 DEBUG { 01:00:46 DEBUG "resultType": "vector", 01:00:46 DEBUG "result": [ 01:00:46 DEBUG "data": { 01:00:46 DEBUG "resultType": "vector", 01:00:46 DEBUG 01:00:46 DEBUG }, 01:00:46 DEBUG "result": [ 01:00:46 DEBUG "status": "success", 01:00:46 DEBUG "result": [ 01:00:46 DEBUG { 01:00:46 DEBUG "resultType": "vector", 01:00:46 DEBUG "result": [ 01:00:46 DEBUG [...] 01:00:46 DEBUG "value": [ 01:00:46 DEBUG { 01:00:46 DEBUG "data": { 01:00:46 DEBUG { 01:00:46 DEBUG "metric": { 01:00:46 DEBUG "result": [ 01:00:46 DEBUG { 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG "metric": { 01:00:46 DEBUG "resultType": "vector", 01:00:46 DEBUG "metric": { 01:00:46 DEBUG "instance": "10.142.0.107:9100" 01:00:46 DEBUG { 01:00:46 DEBUG "metric": { 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG "pod": "ds-cts-0" 01:00:46 DEBUG "result": [ 01:00:46 DEBUG "instance": "10.142.0.107:9100" 01:00:46 DEBUG }, 01:00:46 DEBUG "metric": { 01:00:46 DEBUG "pod": "ds-cts-0" 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG }, 01:00:46 DEBUG { 01:00:46 DEBUG }, 01:00:46 DEBUG "value": [ 01:00:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:00:46 DEBUG }, 01:00:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:00:46 DEBUG [...] 01:00:46 DEBUG "value": [ 01:00:46 DEBUG "metric": { 01:00:46 DEBUG "value": [ 01:00:46 DEBUG 01:00:46 DEBUG }, 01:00:46 DEBUG "value": [ 01:00:46 DEBUG }, 01:00:46 DEBUG "value": [ 01:00:46 DEBUG 01:00:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG "value": [ 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG 1713484846, 01:00:46 DEBUG 01:00:46 DEBUG }, 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG "20.076" 01:00:46 DEBUG 01:00:46 DEBUG "value": [ 01:00:46 DEBUG 01:00:46 DEBUG [...] 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG ] 01:00:46 DEBUG [...] 01:00:46 DEBUG 01:00:46 DEBUG [...] 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG [...] 01:00:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:00:46 DEBUG } 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG [...] 01:00:46 DEBUG 01:00:46 DEBUG }, 01:00:46 DEBUG ] 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG "value": [ 01:00:46 DEBUG } 01:00:46 DEBUG 01:00:46 DEBUG [...] 01:00:46 DEBUG 01:00:46 DEBUG "instance": "10.142.0.43:9100" 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG 1713484846, 01:00:46 DEBUG } 01:00:46 DEBUG "pod": "ds-cts-2" 01:00:46 DEBUG 01:00:46 DEBUG }, 01:00:46 DEBUG 01:00:46 DEBUG 01:00:46 DEBUG "pod": "ds-cts-2" 01:00:46 DEBUG "1919.8" 01:00:46 DEBUG }, 01:00:46 DEBUG 01:00:46 DEBUG "value": [ 01:00:46 DEBUG "instance": "10.142.0.43:9100" 01:00:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:00:46 DEBUG }, 01:00:46 DEBUG ] 01:00:46 DEBUG "value": [ 01:00:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:00:46 DEBUG 1713484846, 01:00:46 DEBUG }, 01:00:46 DEBUG }, 01:00:46 DEBUG "value": [ 01:00:46 DEBUG } 01:00:46 DEBUG ] 01:00:46 DEBUG }, 01:00:46 DEBUG "47363593.96666667" 01:00:46 DEBUG "value": [ 01:00:46 DEBUG "value": [ 01:00:46 DEBUG 1713484846, 01:00:46 DEBUG 1713484846, 01:00:46 DEBUG "0.000033333333333333335" 01:00:46 DEBUG "value": [ 01:00:46 DEBUG ] 01:00:46 DEBUG 1713484846, 01:00:46 DEBUG 1713484846, 01:00:46 DEBUG "1162.9333333333334" 01:00:46 DEBUG ] 01:00:46 DEBUG } 01:00:46 DEBUG 1713484846, 01:00:46 DEBUG } 01:00:46 DEBUG "1.1613333333333324" 01:00:46 DEBUG "1063448576" 01:00:46 DEBUG ] 01:00:46 DEBUG } 01:00:46 DEBUG } 01:00:46 DEBUG "1955574" 01:00:46 DEBUG ] 01:00:46 DEBUG ] 01:00:46 DEBUG ] 01:00:46 DEBUG } 01:00:46 DEBUG ] 01:00:46 DEBUG ] 01:00:46 DEBUG } 01:00:46 DEBUG } 01:00:46 DEBUG } 01:00:46 DEBUG ] 01:00:46 DEBUG } 01:00:46 DEBUG } 01:00:46 DEBUG } 01:00:46 DEBUG ] 01:00:46 DEBUG ] 01:00:46 DEBUG } 01:00:46 DEBUG } 01:00:46 DEBUG ] 01:00:46 DEBUG } 01:00:46 DEBUG } 01:00:46 DEBUG } 01:00:46 DEBUG } 01:00:46 DEBUG } 01:00:46 DEBUG } 01:00:46 DEBUG } 01:00:46 INFO [loop_until]: OK (rc = 0) 01:00:46 DEBUG --- stdout --- 01:00:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10664m 9385Mi am-6fd554f8f7-hvvkw 8489m 8873Mi am-6fd554f8f7-pmj4b 9314m 9028Mi ds-cts-0 9970m 12317Mi ds-cts-1 8864m 12296Mi ds-cts-2 7886m 12365Mi ds-idrepo-0 4788m 3257Mi ds-idrepo-1 4589m 3105Mi ds-idrepo-2 4515m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 4m 1215Mi lodemon-5d46798758-5h2st 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6961m 923Mi 01:00:46 DEBUG --- stderr --- 01:00:46 DEBUG 01:00:46 INFO 01:00:46 INFO [loop_until]: kubectl --namespace=xlou top node 01:00:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:00:46 INFO [loop_until]: OK (rc = 0) 01:00:46 DEBUG --- stdout --- 01:00:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5936Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4638m 15% 6653Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8193m 27% 11601Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3232Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9222m 30% 14797Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 167m 0% 5214Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10511m 35% 12167Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28065m 94% 3126Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9748m 32% 11252Mi 10% gke-xlou-cdm-ds-0f601751-j488 5005m 16% 6076Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8795m 29% 16299Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5292m 17% 6739Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 7994m 26% 14257Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7094m 44% 4670Mi 7% 01:00:46 DEBUG --- stderr --- 01:00:46 DEBUG 01:01:44 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484904" 01:01:45 INFO [http_cmd]: http status code OK 01:01:45 DEBUG --- status code --- 01:01:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:45 DEBUG --- http response --- 01:01:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:01:45 DEBUG ----- output ----- 01:01:45 DEBUG { 01:01:45 DEBUG "status": "success", 01:01:45 DEBUG "data": { 01:01:45 DEBUG "resultType": "vector", 01:01:45 DEBUG "result": [ 01:01:45 DEBUG { 01:01:45 DEBUG "metric": { 01:01:45 DEBUG "instance": "10.142.0.107:9100" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG [...] 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG "instance": "10.142.0.43:9100" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 1713484904, 01:01:45 DEBUG "0.964299999999999" 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG } 01:01:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484905" 01:01:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484905" 01:01:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484905" 01:01:45 ERROR [http_cmd]: ERROR 01:01:45 ERROR --- status code --- 01:01:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:01:45 ERROR --- http response --- 01:01:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:01:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:01:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484905" 01:01:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484905" 01:01:45 INFO [http_cmd]: http status code OK 01:01:45 INFO [http_cmd]: http status code OK 01:01:45 DEBUG --- status code --- 01:01:45 DEBUG --- status code --- 01:01:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:45 DEBUG --- http response --- 01:01:45 DEBUG --- http response --- 01:01:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:01:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:01:45 DEBUG ----- output ----- 01:01:45 DEBUG { 01:01:45 DEBUG "status": "success", 01:01:45 DEBUG "data": { 01:01:45 DEBUG "resultType": "vector", 01:01:45 DEBUG "result": [ 01:01:45 DEBUG { 01:01:45 DEBUG "metric": { 01:01:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG [...] 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 1713484905, 01:01:45 DEBUG "516.7965176797148" 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG } 01:01:45 INFO [http_cmd]: http status code OK 01:01:45 DEBUG --- status code --- 01:01:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:45 DEBUG --- http response --- 01:01:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:01:45 DEBUG ----- output ----- 01:01:45 DEBUG { 01:01:45 DEBUG "status": "success", 01:01:45 DEBUG "data": { 01:01:45 DEBUG "resultType": "vector", 01:01:45 DEBUG "result": [ 01:01:45 DEBUG { 01:01:45 DEBUG "metric": { 01:01:45 DEBUG "instance": "10.142.0.107:9100" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG [...] 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG "instance": "10.142.0.43:9100" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 1713484905, 01:01:45 DEBUG "314682209280" 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG } 01:01:45 INFO [http_cmd]: http status code OK 01:01:45 DEBUG --- status code --- 01:01:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:45 DEBUG --- http response --- 01:01:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:45 DEBUG ----- output ----- 01:01:45 DEBUG { 01:01:45 DEBUG "status": "success", 01:01:45 DEBUG "data": { 01:01:45 DEBUG "resultType": "vector", 01:01:45 DEBUG "result": [ 01:01:45 DEBUG { 01:01:45 DEBUG "metric": { 01:01:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG [...] 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 1713484905, 01:01:45 DEBUG "0" 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG } 01:01:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484905" 01:01:45 INFO [http_cmd]: http status code OK 01:01:45 DEBUG --- status code --- 01:01:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:45 DEBUG --- http response --- 01:01:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:01:45 DEBUG ----- output ----- 01:01:45 DEBUG { 01:01:45 DEBUG "status": "success", 01:01:45 DEBUG "data": { 01:01:45 DEBUG "resultType": "vector", 01:01:45 DEBUG "result": [ 01:01:45 DEBUG { 01:01:45 DEBUG "metric": { 01:01:45 DEBUG "pod": "ds-idrepo-0" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG [...] 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG "pod": "ds-cts-2" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 1713484905, 01:01:45 DEBUG "0" 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG } 01:01:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484905" 01:01:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484905" 01:01:45 INFO [http_cmd]: http status code OK 01:01:45 DEBUG --- status code --- 01:01:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:45 DEBUG --- http response --- 01:01:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:01:45 DEBUG ----- output ----- 01:01:45 DEBUG { 01:01:45 DEBUG "status": "success", 01:01:45 DEBUG "data": { 01:01:45 DEBUG "resultType": "vector", 01:01:45 DEBUG "result": [ 01:01:45 DEBUG { 01:01:45 DEBUG "metric": { 01:01:45 DEBUG "instance": "10.142.0.107:9100" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG [...] 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG "instance": "10.142.0.43:9100" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 1713484905, 01:01:45 DEBUG "46736010.9" 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG } 01:01:45 INFO [http_cmd]: http status code OK 01:01:45 DEBUG --- status code --- 01:01:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:45 DEBUG --- http response --- 01:01:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484905" 01:01:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:01:45 DEBUG ----- output ----- 01:01:45 DEBUG { 01:01:45 DEBUG "status": "success", 01:01:45 DEBUG "data": { 01:01:45 DEBUG "resultType": "vector", 01:01:45 DEBUG "result": [ 01:01:45 DEBUG { 01:01:45 DEBUG "metric": { 01:01:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG [...] 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 1713484905, 01:01:45 DEBUG "49354.06743841277" 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG } 01:01:45 INFO [http_cmd]: http status code OK 01:01:45 DEBUG --- status code --- 01:01:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:45 DEBUG --- http response --- 01:01:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:01:45 DEBUG ----- output ----- 01:01:45 DEBUG { 01:01:45 DEBUG "status": "success", 01:01:45 DEBUG "data": { 01:01:45 DEBUG "resultType": "vector", 01:01:45 DEBUG "result": [ 01:01:45 DEBUG { 01:01:45 DEBUG "metric": { 01:01:45 DEBUG "pod": "ds-cts-0" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG [...] 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG "pod": "ds-cts-2" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 1713484905, 01:01:45 DEBUG "0.6" 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG } 01:01:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484905" 01:01:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484905" 01:01:45 INFO [http_cmd]: http status code OK 01:01:45 DEBUG --- status code --- 01:01:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:45 DEBUG --- http response --- 01:01:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:45 DEBUG ----- output ----- 01:01:45 DEBUG { 01:01:45 DEBUG "status": "success", 01:01:45 DEBUG "data": { 01:01:45 DEBUG "resultType": "vector", 01:01:45 DEBUG "result": [ 01:01:45 DEBUG { 01:01:45 DEBUG "metric": { 01:01:45 DEBUG "pod": "ds-idrepo-0" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG [...] 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG "pod": "ds-idrepo-2" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 1713484905, 01:01:45 DEBUG "0" 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG } 01:01:45 INFO [http_cmd]: http status code OK 01:01:45 DEBUG --- status code --- 01:01:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:45 DEBUG --- http response --- 01:01:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:45 DEBUG ----- output ----- 01:01:45 DEBUG { 01:01:45 DEBUG "status": "success", 01:01:45 DEBUG "data": { 01:01:45 DEBUG "resultType": "vector", 01:01:45 DEBUG "result": [ 01:01:45 DEBUG { 01:01:45 DEBUG "metric": { 01:01:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG [...] 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 1713484905, 01:01:45 DEBUG "1910.1333333333334" 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG } 01:01:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484905" 01:01:45 INFO [http_cmd]: http status code OK 01:01:45 DEBUG --- status code --- 01:01:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:45 DEBUG --- http response --- 01:01:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:01:45 DEBUG ----- output ----- 01:01:45 DEBUG { 01:01:45 DEBUG "status": "success", 01:01:45 DEBUG "data": { 01:01:45 DEBUG "resultType": "vector", 01:01:45 DEBUG "result": [ 01:01:45 DEBUG { 01:01:45 DEBUG "metric": { 01:01:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG [...] 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 1713484905, 01:01:45 DEBUG "9.20087455445854" 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG } 01:01:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484905" 01:01:45 INFO [http_cmd]: http status code OK 01:01:45 DEBUG --- status code --- 01:01:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:45 DEBUG --- http response --- 01:01:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:01:45 DEBUG ----- output ----- 01:01:45 DEBUG { 01:01:45 DEBUG "status": "success", 01:01:45 DEBUG "data": { 01:01:45 DEBUG "resultType": "vector", 01:01:45 DEBUG "result": [ 01:01:45 DEBUG { 01:01:45 DEBUG "metric": { 01:01:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG [...] 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 1713484905, 01:01:45 DEBUG "6.954543648651486" 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG } 01:01:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484905" 01:01:45 INFO [http_cmd]: http status code OK 01:01:45 DEBUG --- status code --- 01:01:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:45 DEBUG --- http response --- 01:01:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:45 DEBUG ----- output ----- 01:01:45 DEBUG { 01:01:45 DEBUG "status": "success", 01:01:45 DEBUG "data": { 01:01:45 DEBUG "resultType": "vector", 01:01:45 DEBUG "result": [ 01:01:45 DEBUG { 01:01:45 DEBUG "metric": { 01:01:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG [...] 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG 01:01:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:01:45 DEBUG }, 01:01:45 DEBUG "value": [ 01:01:45 DEBUG 1713484905, 01:01:45 DEBUG "0" 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG ] 01:01:45 DEBUG } 01:01:45 DEBUG } 01:01:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484906" 01:01:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484906" 01:01:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484906" 01:01:46 INFO [http_cmd]: http status code OK 01:01:46 INFO [http_cmd]: http status code OK 01:01:46 DEBUG --- status code --- 01:01:46 DEBUG --- status code --- 01:01:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:46 DEBUG --- http response --- 01:01:46 DEBUG --- http response --- 01:01:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:01:46 DEBUG ----- output ----- 01:01:46 DEBUG ----- output ----- 01:01:46 DEBUG { 01:01:46 DEBUG { 01:01:46 DEBUG "status": "success", 01:01:46 DEBUG "status": "success", 01:01:46 DEBUG "data": { 01:01:46 DEBUG "data": { 01:01:46 DEBUG "resultType": "vector", 01:01:46 DEBUG "resultType": "vector", 01:01:46 INFO [http_cmd]: http status code OK 01:01:46 DEBUG "result": [ 01:01:46 DEBUG "result": [ 01:01:46 DEBUG --- status code --- 01:01:46 DEBUG { 01:01:46 DEBUG { 01:01:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:46 DEBUG --- http response --- 01:01:46 DEBUG "metric": { 01:01:46 DEBUG "metric": { 01:01:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:01:46 DEBUG "pod": "ds-idrepo-0" 01:01:46 DEBUG ----- output ----- 01:01:46 DEBUG }, 01:01:46 DEBUG }, 01:01:46 DEBUG { 01:01:46 DEBUG "value": [ 01:01:46 DEBUG "value": [ 01:01:46 DEBUG "status": "success", 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG "data": { 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG "resultType": "vector", 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG "result": [ 01:01:46 DEBUG [...] 01:01:46 DEBUG [...] 01:01:46 DEBUG { 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG "metric": { 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG "pod": "ds-idrepo-0" 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG }, 01:01:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:01:46 DEBUG "pod": "ds-idrepo-2" 01:01:46 DEBUG "value": [ 01:01:46 DEBUG }, 01:01:46 DEBUG }, 01:01:46 DEBUG 01:01:46 DEBUG "value": [ 01:01:46 DEBUG "value": [ 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG 1713484906, 01:01:46 DEBUG 1713484906, 01:01:46 DEBUG [...] 01:01:46 DEBUG "0" 01:01:46 DEBUG "0" 01:01:46 DEBUG 01:01:46 DEBUG ] 01:01:46 DEBUG ] 01:01:46 DEBUG 01:01:46 DEBUG } 01:01:46 DEBUG } 01:01:46 DEBUG 01:01:46 DEBUG ] 01:01:46 DEBUG ] 01:01:46 DEBUG "pod": "ds-idrepo-2" 01:01:46 DEBUG } 01:01:46 DEBUG } 01:01:46 DEBUG }, 01:01:46 DEBUG } 01:01:46 DEBUG } 01:01:46 DEBUG "value": [ 01:01:46 DEBUG 1713484906, 01:01:46 DEBUG "0" 01:01:46 DEBUG ] 01:01:46 DEBUG } 01:01:46 DEBUG ] 01:01:46 DEBUG } 01:01:46 DEBUG } 01:01:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484906" 01:01:46 INFO [http_cmd]: http status code OK 01:01:46 DEBUG --- status code --- 01:01:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:46 DEBUG --- http response --- 01:01:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:46 DEBUG ----- output ----- 01:01:46 DEBUG { 01:01:46 DEBUG "status": "success", 01:01:46 DEBUG "data": { 01:01:46 DEBUG "resultType": "vector", 01:01:46 DEBUG "result": [ 01:01:46 DEBUG { 01:01:46 DEBUG "metric": { 01:01:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:01:46 DEBUG }, 01:01:46 DEBUG "value": [ 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG [...] 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:01:46 DEBUG }, 01:01:46 DEBUG "value": [ 01:01:46 DEBUG 1713484906, 01:01:46 DEBUG "1910.0666666666666" 01:01:46 DEBUG ] 01:01:46 DEBUG } 01:01:46 DEBUG ] 01:01:46 DEBUG } 01:01:46 DEBUG } 01:01:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484906" 01:01:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484906" 01:01:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484906" 01:01:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484906" 01:01:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484906" 01:01:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484906" 01:01:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484906" 01:01:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484906" 01:01:46 INFO [http_cmd]: http status code OK 01:01:46 DEBUG --- status code --- 01:01:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:46 DEBUG --- http response --- 01:01:46 INFO [http_cmd]: http status code OK 01:01:46 INFO [http_cmd]: http status code OK 01:01:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:46 DEBUG --- status code --- 01:01:46 DEBUG --- status code --- 01:01:46 INFO [http_cmd]: http status code OK 01:01:46 DEBUG ----- output ----- 01:01:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:46 DEBUG --- status code --- 01:01:46 DEBUG { 01:01:46 DEBUG --- http response --- 01:01:46 DEBUG --- http response --- 01:01:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:46 INFO [http_cmd]: http status code OK 01:01:46 DEBUG "status": "success", 01:01:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:01:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:46 DEBUG --- http response --- 01:01:46 DEBUG --- status code --- 01:01:46 DEBUG "data": { 01:01:46 DEBUG ----- output ----- 01:01:46 DEBUG ----- output ----- 01:01:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:01:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:46 INFO [http_cmd]: http status code OK 01:01:46 DEBUG "resultType": "vector", 01:01:46 DEBUG { 01:01:46 DEBUG { 01:01:46 DEBUG ----- output ----- 01:01:46 DEBUG --- http response --- 01:01:46 DEBUG --- status code --- 01:01:46 DEBUG "result": [ 01:01:46 DEBUG "status": "success", 01:01:46 DEBUG "status": "success", 01:01:46 DEBUG { 01:01:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:01:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:46 DEBUG { 01:01:46 DEBUG "data": { 01:01:46 DEBUG "data": { 01:01:46 DEBUG "status": "success", 01:01:46 DEBUG ----- output ----- 01:01:46 DEBUG --- http response --- 01:01:46 DEBUG "metric": { 01:01:46 DEBUG "resultType": "vector", 01:01:46 DEBUG "resultType": "vector", 01:01:46 DEBUG "data": { 01:01:46 DEBUG { 01:01:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:01:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:01:46 INFO [http_cmd]: http status code OK 01:01:46 DEBUG "result": [ 01:01:46 DEBUG "result": [ 01:01:46 DEBUG "resultType": "vector", 01:01:46 DEBUG "status": "success", 01:01:46 DEBUG }, 01:01:46 DEBUG ----- output ----- 01:01:46 DEBUG --- status code --- 01:01:46 DEBUG { 01:01:46 INFO [http_cmd]: http status code OK 01:01:46 DEBUG { 01:01:46 DEBUG "result": [ 01:01:46 DEBUG "data": { 01:01:46 DEBUG "value": [ 01:01:46 DEBUG { 01:01:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:46 DEBUG "metric": { 01:01:46 DEBUG --- status code --- 01:01:46 DEBUG "metric": { 01:01:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:01:46 DEBUG "resultType": "vector", 01:01:46 DEBUG 01:01:46 DEBUG "status": "success", 01:01:46 DEBUG --- http response --- 01:01:46 DEBUG "instance": "10.142.0.107:9100" 01:01:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:01:46 DEBUG { 01:01:46 DEBUG }, 01:01:46 DEBUG "result": [ 01:01:46 DEBUG 01:01:46 DEBUG "data": { 01:01:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:01:46 DEBUG }, 01:01:46 DEBUG --- http response --- 01:01:46 DEBUG "metric": { 01:01:46 DEBUG "value": [ 01:01:46 DEBUG { 01:01:46 DEBUG 01:01:46 DEBUG "resultType": "vector", 01:01:46 DEBUG ----- output ----- 01:01:46 DEBUG "value": [ 01:01:46 DEBUG "pod": "ds-cts-0" 01:01:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:01:46 DEBUG 01:01:46 DEBUG "metric": { 01:01:46 DEBUG [...] 01:01:46 DEBUG "result": [ 01:01:46 DEBUG { 01:01:46 DEBUG 01:01:46 DEBUG }, 01:01:46 DEBUG ----- output ----- 01:01:46 DEBUG 01:01:46 DEBUG "pod": "ds-cts-0" 01:01:46 DEBUG 01:01:46 DEBUG { 01:01:46 DEBUG "status": "success", 01:01:46 DEBUG 01:01:46 DEBUG "value": [ 01:01:46 DEBUG { 01:01:46 DEBUG 01:01:46 DEBUG }, 01:01:46 DEBUG 01:01:46 DEBUG "metric": { 01:01:46 DEBUG "data": { 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG "status": "success", 01:01:46 DEBUG [...] 01:01:46 DEBUG "value": [ 01:01:46 DEBUG 01:01:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:01:46 DEBUG "resultType": "vector", 01:01:46 DEBUG [...] 01:01:46 DEBUG 01:01:46 DEBUG "data": { 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:01:46 DEBUG }, 01:01:46 DEBUG "result": [ 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG "resultType": "vector", 01:01:46 DEBUG "result": [ 01:01:46 DEBUG 01:01:46 DEBUG }, 01:01:46 DEBUG "value": [ 01:01:46 DEBUG { 01:01:46 DEBUG 01:01:46 DEBUG [...] 01:01:46 DEBUG { 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG "value": [ 01:01:46 DEBUG 01:01:46 DEBUG "metric": { 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG "metric": { 01:01:46 DEBUG 01:01:46 DEBUG [...] 01:01:46 DEBUG 1713484906, 01:01:46 DEBUG 01:01:46 DEBUG "instance": "10.142.0.107:9100" 01:01:46 DEBUG "instance": "10.142.0.43:9100" 01:01:46 DEBUG 01:01:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:01:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:01:46 DEBUG }, 01:01:46 DEBUG "20.076" 01:01:46 DEBUG 01:01:46 DEBUG }, 01:01:46 DEBUG }, 01:01:46 DEBUG 01:01:46 DEBUG }, 01:01:46 DEBUG 01:01:46 DEBUG "value": [ 01:01:46 DEBUG 1713484906, 01:01:46 DEBUG [...] 01:01:46 DEBUG "value": [ 01:01:46 DEBUG "value": [ 01:01:46 DEBUG "pod": "ds-cts-2" 01:01:46 DEBUG "value": [ 01:01:46 DEBUG 01:01:46 DEBUG "1909.9666666666667" 01:01:46 DEBUG ] 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG 1713484906, 01:01:46 DEBUG }, 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG ] 01:01:46 DEBUG } 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG "46741180.13333333" 01:01:46 DEBUG "value": [ 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG } 01:01:46 DEBUG ] 01:01:46 DEBUG 01:01:46 DEBUG 01:01:46 DEBUG ] 01:01:46 DEBUG } 01:01:46 DEBUG [...] 01:01:46 DEBUG "pod": "ds-cts-2" 01:01:46 DEBUG ] 01:01:46 DEBUG } 01:01:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:01:46 DEBUG [...] 01:01:46 DEBUG 1713484906, 01:01:46 DEBUG ] 01:01:46 DEBUG 01:01:46 DEBUG }, 01:01:46 DEBUG } 01:01:46 DEBUG } 01:01:46 DEBUG }, 01:01:46 DEBUG 01:01:46 DEBUG "0.00032659999999999997" 01:01:46 DEBUG } 01:01:46 DEBUG 01:01:46 DEBUG "value": [ 01:01:46 DEBUG } 01:01:46 DEBUG "value": [ 01:01:46 DEBUG 01:01:46 DEBUG ] 01:01:46 DEBUG } 01:01:46 DEBUG 01:01:46 DEBUG 1713484906, 01:01:46 DEBUG 1713484906, 01:01:46 DEBUG 01:01:46 DEBUG } 01:01:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:01:46 DEBUG "1743.1333333333332" 01:01:46 DEBUG ] 01:01:46 DEBUG "instance": "10.142.0.43:9100" 01:01:46 DEBUG ] 01:01:46 DEBUG } 01:01:46 DEBUG } 01:01:46 DEBUG } 01:01:46 DEBUG }, 01:01:46 DEBUG }, 01:01:46 DEBUG "1065201664" 01:01:46 DEBUG ] 01:01:46 DEBUG "value": [ 01:01:46 DEBUG "value": [ 01:01:46 DEBUG ] 01:01:46 DEBUG } 01:01:46 DEBUG 1713484906, 01:01:46 DEBUG 1713484906, 01:01:46 DEBUG } 01:01:46 DEBUG } 01:01:46 DEBUG "1.2299999999999995" 01:01:46 DEBUG "2070484.5" 01:01:46 DEBUG ] 01:01:46 DEBUG ] 01:01:46 DEBUG } 01:01:46 DEBUG } 01:01:46 DEBUG ] 01:01:46 DEBUG ] 01:01:46 DEBUG } 01:01:46 DEBUG } 01:01:46 DEBUG } 01:01:46 DEBUG } 01:01:46 DEBUG ] 01:01:46 DEBUG } 01:01:46 DEBUG } 01:01:46 INFO 01:01:46 INFO [loop_until]: kubectl --namespace=xlou top pods 01:01:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:01:46 INFO [loop_until]: OK (rc = 0) 01:01:46 DEBUG --- stdout --- 01:01:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10642m 9386Mi am-6fd554f8f7-hvvkw 8388m 8875Mi am-6fd554f8f7-pmj4b 9306m 9029Mi ds-cts-0 8872m 12296Mi ds-cts-1 8933m 12285Mi ds-cts-2 8468m 12403Mi ds-idrepo-0 4831m 3257Mi ds-idrepo-1 4793m 3122Mi ds-idrepo-2 4580m 3038Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6880m 924Mi 01:01:46 DEBUG --- stderr --- 01:01:46 DEBUG 01:01:46 INFO 01:01:46 INFO [loop_until]: kubectl --namespace=xlou top node 01:01:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:01:46 INFO [loop_until]: OK (rc = 0) 01:01:46 DEBUG --- stdout --- 01:01:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5913Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4620m 15% 6661Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8272m 27% 11602Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3234Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8927m 29% 14764Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5194Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10500m 35% 12179Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27670m 92% 3128Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9947m 33% 11259Mi 10% gke-xlou-cdm-ds-0f601751-j488 4974m 16% 6081Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9008m 30% 16313Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5252m 17% 6746Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8525m 28% 14306Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7002m 44% 4668Mi 7% 01:01:46 DEBUG --- stderr --- 01:01:46 DEBUG 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484965" 01:02:45 INFO [http_cmd]: http status code OK 01:02:45 DEBUG --- status code --- 01:02:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:45 DEBUG --- http response --- 01:02:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:45 DEBUG ----- output ----- 01:02:45 DEBUG { 01:02:45 DEBUG "status": "success", 01:02:45 DEBUG "data": { 01:02:45 DEBUG "resultType": "vector", 01:02:45 DEBUG "result": [ 01:02:45 DEBUG { 01:02:45 DEBUG "metric": { 01:02:45 DEBUG "instance": "10.142.0.107:9100" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG [...] 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG "instance": "10.142.0.43:9100" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 1713484965, 01:02:45 DEBUG "0.9725999999999989" 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG } 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713484965" 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484965" 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484965" 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484965" 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484965" 01:02:45 ERROR [http_cmd]: ERROR 01:02:45 ERROR --- status code --- 01:02:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:02:45 ERROR --- http response --- 01:02:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:02:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:02:45 INFO [http_cmd]: http status code OK 01:02:45 INFO [http_cmd]: http status code OK 01:02:45 DEBUG --- status code --- 01:02:45 DEBUG --- status code --- 01:02:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:45 DEBUG --- http response --- 01:02:45 DEBUG --- http response --- 01:02:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:02:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:02:45 DEBUG ----- output ----- 01:02:45 DEBUG { 01:02:45 DEBUG "status": "success", 01:02:45 DEBUG "data": { 01:02:45 DEBUG "resultType": "vector", 01:02:45 DEBUG "result": [ 01:02:45 DEBUG { 01:02:45 DEBUG "metric": { 01:02:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG [...] 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 1713484965, 01:02:45 DEBUG "521.1030183518336" 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG } 01:02:45 INFO [http_cmd]: http status code OK 01:02:45 INFO [http_cmd]: http status code OK 01:02:45 DEBUG --- status code --- 01:02:45 DEBUG --- status code --- 01:02:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:45 DEBUG --- http response --- 01:02:45 DEBUG --- http response --- 01:02:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:45 DEBUG ----- output ----- 01:02:45 DEBUG { 01:02:45 DEBUG "status": "success", 01:02:45 DEBUG "data": { 01:02:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:45 DEBUG "resultType": "vector", 01:02:45 DEBUG ----- output ----- 01:02:45 DEBUG "result": [ 01:02:45 DEBUG { 01:02:45 DEBUG { 01:02:45 DEBUG "status": "success", 01:02:45 DEBUG "metric": { 01:02:45 DEBUG "data": { 01:02:45 DEBUG "instance": "10.142.0.107:9100" 01:02:45 DEBUG "resultType": "vector", 01:02:45 DEBUG }, 01:02:45 DEBUG "result": [ 01:02:45 DEBUG "value": [ 01:02:45 DEBUG { 01:02:45 DEBUG 01:02:45 DEBUG "metric": { 01:02:45 DEBUG 01:02:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:02:45 DEBUG 01:02:45 DEBUG }, 01:02:45 DEBUG [...] 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG "instance": "10.142.0.43:9100" 01:02:45 DEBUG [...] 01:02:45 DEBUG }, 01:02:45 DEBUG 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 01:02:45 DEBUG 1713484965, 01:02:45 DEBUG 01:02:45 DEBUG "317055458304" 01:02:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:02:45 DEBUG ] 01:02:45 DEBUG }, 01:02:45 DEBUG } 01:02:45 DEBUG "value": [ 01:02:45 DEBUG ] 01:02:45 DEBUG 1713484965, 01:02:45 DEBUG } 01:02:45 DEBUG "0" 01:02:45 DEBUG } 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG } 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713484965" 01:02:45 INFO [http_cmd]: http status code OK 01:02:45 DEBUG --- status code --- 01:02:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:45 DEBUG --- http response --- 01:02:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:02:45 DEBUG ----- output ----- 01:02:45 DEBUG { 01:02:45 DEBUG "status": "success", 01:02:45 DEBUG "data": { 01:02:45 DEBUG "resultType": "vector", 01:02:45 DEBUG "result": [ 01:02:45 DEBUG { 01:02:45 DEBUG "metric": { 01:02:45 DEBUG "pod": "ds-idrepo-0" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG [...] 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG "pod": "ds-cts-2" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 1713484965, 01:02:45 DEBUG "0" 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG } 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484965" 01:02:45 INFO [http_cmd]: http status code OK 01:02:45 DEBUG --- status code --- 01:02:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:45 DEBUG --- http response --- 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484965" 01:02:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:45 DEBUG ----- output ----- 01:02:45 DEBUG { 01:02:45 DEBUG "status": "success", 01:02:45 DEBUG "data": { 01:02:45 DEBUG "resultType": "vector", 01:02:45 DEBUG "result": [ 01:02:45 DEBUG { 01:02:45 DEBUG "metric": { 01:02:45 DEBUG "instance": "10.142.0.107:9100" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG [...] 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG "instance": "10.142.0.43:9100" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 1713484965, 01:02:45 DEBUG "46940118.33333333" 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG } 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484965" 01:02:45 INFO [http_cmd]: http status code OK 01:02:45 DEBUG --- status code --- 01:02:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:45 DEBUG --- http response --- 01:02:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:02:45 DEBUG ----- output ----- 01:02:45 DEBUG { 01:02:45 DEBUG "status": "success", 01:02:45 DEBUG "data": { 01:02:45 DEBUG "resultType": "vector", 01:02:45 DEBUG "result": [ 01:02:45 DEBUG { 01:02:45 DEBUG "metric": { 01:02:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG [...] 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 1713484965, 01:02:45 DEBUG "11985.369422092173" 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG } 01:02:45 INFO [http_cmd]: http status code OK 01:02:45 DEBUG --- status code --- 01:02:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:45 DEBUG --- http response --- 01:02:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:02:45 DEBUG ----- output ----- 01:02:45 DEBUG { 01:02:45 DEBUG "status": "success", 01:02:45 DEBUG "data": { 01:02:45 DEBUG "resultType": "vector", 01:02:45 DEBUG "result": [ 01:02:45 DEBUG { 01:02:45 DEBUG "metric": { 01:02:45 DEBUG "pod": "ds-cts-0" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG [...] 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG "pod": "ds-cts-2" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 1713484965, 01:02:45 DEBUG "0.5333333333333333" 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG } 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484965" 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484965" 01:02:45 INFO [http_cmd]: http status code OK 01:02:45 INFO [http_cmd]: http status code OK 01:02:45 DEBUG --- status code --- 01:02:45 DEBUG --- status code --- 01:02:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:45 DEBUG --- http response --- 01:02:45 DEBUG --- http response --- 01:02:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:45 DEBUG ----- output ----- 01:02:45 DEBUG ----- output ----- 01:02:45 DEBUG { 01:02:45 DEBUG { 01:02:45 DEBUG "status": "success", 01:02:45 DEBUG "status": "success", 01:02:45 DEBUG "data": { 01:02:45 DEBUG "data": { 01:02:45 DEBUG "resultType": "vector", 01:02:45 DEBUG "resultType": "vector", 01:02:45 DEBUG "result": [ 01:02:45 DEBUG "result": [ 01:02:45 DEBUG { 01:02:45 DEBUG { 01:02:45 DEBUG "metric": { 01:02:45 DEBUG "metric": { 01:02:45 DEBUG "pod": "ds-idrepo-0" 01:02:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:02:45 DEBUG }, 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG [...] 01:02:45 DEBUG [...] 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG "pod": "ds-idrepo-2" 01:02:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:02:45 DEBUG }, 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 1713484965, 01:02:45 DEBUG 1713484965, 01:02:45 DEBUG "0" 01:02:45 DEBUG "1912.3" 01:02:45 DEBUG ] 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG } 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG } 01:02:45 DEBUG } 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484965" 01:02:45 INFO [http_cmd]: http status code OK 01:02:45 DEBUG --- status code --- 01:02:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:45 DEBUG --- http response --- 01:02:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:02:45 DEBUG ----- output ----- 01:02:45 DEBUG { 01:02:45 DEBUG "status": "success", 01:02:45 DEBUG "data": { 01:02:45 DEBUG "resultType": "vector", 01:02:45 DEBUG "result": [ 01:02:45 DEBUG { 01:02:45 DEBUG "metric": { 01:02:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG [...] 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 1713484965, 01:02:45 DEBUG "8.51868588785345" 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG } 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713484965" 01:02:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484965" 01:02:45 INFO [http_cmd]: http status code OK 01:02:45 DEBUG --- status code --- 01:02:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:45 DEBUG --- http response --- 01:02:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:45 DEBUG ----- output ----- 01:02:45 DEBUG { 01:02:45 DEBUG "status": "success", 01:02:45 DEBUG "data": { 01:02:45 DEBUG "resultType": "vector", 01:02:45 DEBUG "result": [ 01:02:45 DEBUG { 01:02:45 DEBUG "metric": { 01:02:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG [...] 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 1713484965, 01:02:45 DEBUG "6.866254851222088" 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG } 01:02:45 INFO [http_cmd]: http status code OK 01:02:45 DEBUG --- status code --- 01:02:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:45 DEBUG --- http response --- 01:02:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:45 DEBUG ----- output ----- 01:02:45 DEBUG { 01:02:45 DEBUG "status": "success", 01:02:45 DEBUG "data": { 01:02:45 DEBUG "resultType": "vector", 01:02:45 DEBUG "result": [ 01:02:45 DEBUG { 01:02:45 DEBUG "metric": { 01:02:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG [...] 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG 01:02:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:02:45 DEBUG }, 01:02:45 DEBUG "value": [ 01:02:45 DEBUG 1713484965, 01:02:45 DEBUG "0" 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG ] 01:02:45 DEBUG } 01:02:45 DEBUG } 01:02:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484966" 01:02:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484966" 01:02:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484966" 01:02:46 INFO [http_cmd]: http status code OK 01:02:46 INFO [http_cmd]: http status code OK 01:02:46 DEBUG --- status code --- 01:02:46 DEBUG --- status code --- 01:02:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:46 DEBUG --- http response --- 01:02:46 DEBUG --- http response --- 01:02:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:46 DEBUG ----- output ----- 01:02:46 DEBUG { 01:02:46 DEBUG "status": "success", 01:02:46 DEBUG "data": { 01:02:46 DEBUG "resultType": "vector", 01:02:46 DEBUG "result": [ 01:02:46 DEBUG { 01:02:46 DEBUG "metric": { 01:02:46 DEBUG "pod": "ds-idrepo-0" 01:02:46 DEBUG }, 01:02:46 DEBUG "value": [ 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG [...] 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG "pod": "ds-idrepo-2" 01:02:46 DEBUG }, 01:02:46 DEBUG "value": [ 01:02:46 DEBUG 1713484966, 01:02:46 DEBUG "0" 01:02:46 DEBUG ] 01:02:46 DEBUG } 01:02:46 DEBUG ] 01:02:46 DEBUG } 01:02:46 DEBUG } 01:02:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:46 DEBUG ----- output ----- 01:02:46 DEBUG { 01:02:46 DEBUG "status": "success", 01:02:46 DEBUG "data": { 01:02:46 DEBUG "resultType": "vector", 01:02:46 DEBUG "result": [ 01:02:46 DEBUG { 01:02:46 DEBUG "metric": { 01:02:46 DEBUG "pod": "ds-idrepo-0" 01:02:46 DEBUG }, 01:02:46 DEBUG "value": [ 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG [...] 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG "pod": "ds-idrepo-2" 01:02:46 DEBUG }, 01:02:46 DEBUG "value": [ 01:02:46 DEBUG 1713484966, 01:02:46 DEBUG "0" 01:02:46 DEBUG ] 01:02:46 DEBUG } 01:02:46 DEBUG ] 01:02:46 DEBUG } 01:02:46 DEBUG } 01:02:46 INFO [http_cmd]: http status code OK 01:02:46 DEBUG --- status code --- 01:02:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:46 DEBUG --- http response --- 01:02:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:02:46 DEBUG ----- output ----- 01:02:46 DEBUG { 01:02:46 DEBUG "status": "success", 01:02:46 DEBUG "data": { 01:02:46 DEBUG "resultType": "vector", 01:02:46 DEBUG "result": [ 01:02:46 DEBUG { 01:02:46 DEBUG "metric": { 01:02:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:02:46 DEBUG }, 01:02:46 DEBUG "value": [ 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG [...] 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:02:46 DEBUG }, 01:02:46 DEBUG "value": [ 01:02:46 DEBUG 1713484966, 01:02:46 DEBUG "0" 01:02:46 DEBUG ] 01:02:46 DEBUG } 01:02:46 DEBUG ] 01:02:46 DEBUG } 01:02:46 DEBUG } 01:02:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484966" 01:02:46 INFO [http_cmd]: http status code OK 01:02:46 DEBUG --- status code --- 01:02:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:46 DEBUG --- http response --- 01:02:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:46 DEBUG ----- output ----- 01:02:46 DEBUG { 01:02:46 DEBUG "status": "success", 01:02:46 DEBUG "data": { 01:02:46 DEBUG "resultType": "vector", 01:02:46 DEBUG "result": [ 01:02:46 DEBUG { 01:02:46 DEBUG "metric": { 01:02:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:02:46 DEBUG }, 01:02:46 DEBUG "value": [ 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG [...] 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:02:46 DEBUG }, 01:02:46 DEBUG "value": [ 01:02:46 DEBUG 1713484966, 01:02:46 DEBUG "1913.1333333333332" 01:02:46 DEBUG ] 01:02:46 DEBUG } 01:02:46 DEBUG ] 01:02:46 DEBUG } 01:02:46 DEBUG } 01:02:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484966" 01:02:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484966" 01:02:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484966" 01:02:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713484966" 01:02:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484966" 01:02:46 INFO [http_cmd]: http status code OK 01:02:46 INFO [http_cmd]: http status code OK 01:02:46 DEBUG --- status code --- 01:02:46 DEBUG --- status code --- 01:02:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713484966" 01:02:46 INFO [http_cmd]: http status code OK 01:02:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713484966" 01:02:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713484966" 01:02:46 DEBUG --- status code --- 01:02:46 DEBUG --- http response --- 01:02:46 DEBUG --- http response --- 01:02:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:02:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:46 DEBUG --- http response --- 01:02:46 DEBUG ----- output ----- 01:02:46 DEBUG ----- output ----- 01:02:46 INFO [http_cmd]: http status code OK 01:02:46 DEBUG { 01:02:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:46 DEBUG { 01:02:46 DEBUG --- status code --- 01:02:46 DEBUG "status": "success", 01:02:46 DEBUG ----- output ----- 01:02:46 DEBUG "status": "success", 01:02:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:46 DEBUG "data": { 01:02:46 DEBUG { 01:02:46 INFO [http_cmd]: http status code OK 01:02:46 INFO [http_cmd]: http status code OK 01:02:46 DEBUG "data": { 01:02:46 DEBUG --- http response --- 01:02:46 DEBUG "resultType": "vector", 01:02:46 DEBUG "status": "success", 01:02:46 DEBUG --- status code --- 01:02:46 DEBUG --- status code --- 01:02:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:46 DEBUG "result": [ 01:02:46 DEBUG "data": { 01:02:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:46 DEBUG "resultType": "vector", 01:02:46 DEBUG --- http response --- 01:02:46 DEBUG ----- output ----- 01:02:46 DEBUG { 01:02:46 DEBUG "resultType": "vector", 01:02:46 DEBUG "result": [ 01:02:46 DEBUG "result": [ 01:02:46 DEBUG { 01:02:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:02:46 DEBUG "metric": { 01:02:46 DEBUG --- http response --- 01:02:46 DEBUG { 01:02:46 DEBUG { 01:02:46 DEBUG "status": "success", 01:02:46 DEBUG ----- output ----- 01:02:46 DEBUG "pod": "ds-cts-0" 01:02:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:02:46 DEBUG "metric": { 01:02:46 DEBUG "metric": { 01:02:46 DEBUG "data": { 01:02:46 INFO [http_cmd]: http status code OK 01:02:46 DEBUG { 01:02:46 DEBUG }, 01:02:46 DEBUG ----- output ----- 01:02:46 DEBUG "instance": "10.142.0.107:9100" 01:02:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:02:46 DEBUG "resultType": "vector", 01:02:46 DEBUG --- status code --- 01:02:46 DEBUG "status": "success", 01:02:46 DEBUG "value": [ 01:02:46 DEBUG { 01:02:46 DEBUG }, 01:02:46 DEBUG }, 01:02:46 DEBUG "result": [ 01:02:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:46 DEBUG "data": { 01:02:46 DEBUG 01:02:46 INFO [http_cmd]: http status code OK 01:02:46 DEBUG "status": "success", 01:02:46 DEBUG "value": [ 01:02:46 DEBUG "value": [ 01:02:46 DEBUG { 01:02:46 DEBUG --- http response --- 01:02:46 DEBUG "resultType": "vector", 01:02:46 DEBUG 01:02:46 DEBUG --- status code --- 01:02:46 DEBUG "data": { 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG "metric": { 01:02:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:46 DEBUG "result": [ 01:02:46 DEBUG 01:02:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:02:46 DEBUG "resultType": "vector", 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:02:46 DEBUG ----- output ----- 01:02:46 DEBUG { 01:02:46 DEBUG [...] 01:02:46 DEBUG --- http response --- 01:02:46 DEBUG "result": [ 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG }, 01:02:46 DEBUG { 01:02:46 DEBUG "metric": { 01:02:46 DEBUG 01:02:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:02:46 DEBUG { 01:02:46 DEBUG [...] 01:02:46 DEBUG [...] 01:02:46 DEBUG "value": [ 01:02:46 DEBUG "status": "success", 01:02:46 DEBUG "pod": "ds-cts-0" 01:02:46 DEBUG 01:02:46 DEBUG ----- output ----- 01:02:46 DEBUG "metric": { 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG "data": { 01:02:46 DEBUG "resultType": "vector", 01:02:46 DEBUG }, 01:02:46 DEBUG { 01:02:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG "result": [ 01:02:46 DEBUG "value": [ 01:02:46 DEBUG "status": "success", 01:02:46 DEBUG }, 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG "pod": "ds-cts-2" 01:02:46 DEBUG { 01:02:46 DEBUG 01:02:46 DEBUG "data": { 01:02:46 DEBUG "value": [ 01:02:46 DEBUG "instance": "10.142.0.43:9100" 01:02:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:02:46 DEBUG [...] 01:02:46 DEBUG }, 01:02:46 DEBUG "metric": { 01:02:46 DEBUG 01:02:46 DEBUG "resultType": "vector", 01:02:46 DEBUG 01:02:46 DEBUG }, 01:02:46 DEBUG }, 01:02:46 DEBUG 01:02:46 DEBUG "value": [ 01:02:46 DEBUG "instance": "10.142.0.107:9100" 01:02:46 DEBUG 01:02:46 DEBUG "result": [ 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG "value": [ 01:02:46 DEBUG 01:02:46 DEBUG 1713484966, 01:02:46 DEBUG }, 01:02:46 DEBUG [...] 01:02:46 DEBUG { 01:02:46 DEBUG "value": [ 01:02:46 DEBUG [...] 01:02:46 DEBUG 1713484966, 01:02:46 DEBUG 01:02:46 DEBUG "0.00005865333333333333" 01:02:46 DEBUG "value": [ 01:02:46 DEBUG 01:02:46 DEBUG "metric": { 01:02:46 DEBUG 1713484966, 01:02:46 DEBUG 01:02:46 DEBUG "2185038.5" 01:02:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:02:46 DEBUG ] 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:02:46 DEBUG "46942990" 01:02:46 DEBUG 01:02:46 DEBUG ] 01:02:46 DEBUG }, 01:02:46 DEBUG } 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG }, 01:02:46 DEBUG ] 01:02:46 DEBUG 01:02:46 DEBUG } 01:02:46 DEBUG "value": [ 01:02:46 DEBUG ] 01:02:46 DEBUG 01:02:46 DEBUG "pod": "ds-cts-2" 01:02:46 DEBUG "value": [ 01:02:46 DEBUG } 01:02:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:02:46 DEBUG ] 01:02:46 DEBUG 1713484966, 01:02:46 DEBUG } 01:02:46 DEBUG [...] 01:02:46 DEBUG }, 01:02:46 DEBUG 01:02:46 DEBUG ] 01:02:46 DEBUG }, 01:02:46 DEBUG } 01:02:46 DEBUG "20.076" 01:02:46 DEBUG } 01:02:46 DEBUG 01:02:46 DEBUG "value": [ 01:02:46 DEBUG 01:02:46 DEBUG } 01:02:46 DEBUG "value": [ 01:02:46 DEBUG } 01:02:46 DEBUG ] 01:02:46 DEBUG 01:02:46 DEBUG 1713484966, 01:02:46 DEBUG 01:02:46 DEBUG } 01:02:46 DEBUG 1713484966, 01:02:46 DEBUG } 01:02:46 DEBUG 01:02:46 DEBUG "2397.8" 01:02:46 DEBUG [...] 01:02:46 DEBUG "1912.2666666666667" 01:02:46 DEBUG ] 01:02:46 DEBUG "instance": "10.142.0.43:9100" 01:02:46 DEBUG ] 01:02:46 DEBUG 01:02:46 DEBUG 01:02:46 DEBUG } 01:02:46 DEBUG }, 01:02:46 DEBUG } 01:02:46 DEBUG ] 01:02:46 DEBUG 01:02:46 DEBUG } 01:02:46 DEBUG "value": [ 01:02:46 DEBUG ] 01:02:46 DEBUG } 01:02:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:02:46 DEBUG 1713484966, 01:02:46 DEBUG } 01:02:46 DEBUG ] 01:02:46 DEBUG }, 01:02:46 DEBUG "1.3986666666666685" 01:02:46 DEBUG } 01:02:46 DEBUG } 01:02:46 DEBUG "value": [ 01:02:46 DEBUG ] 01:02:46 DEBUG } 01:02:46 DEBUG 1713484966, 01:02:46 DEBUG } 01:02:46 DEBUG "1066557440" 01:02:46 DEBUG ] 01:02:46 DEBUG ] 01:02:46 DEBUG } 01:02:46 DEBUG } 01:02:46 DEBUG ] 01:02:46 DEBUG } 01:02:46 DEBUG } 01:02:46 DEBUG } 01:02:46 INFO 01:02:46 INFO [loop_until]: kubectl --namespace=xlou top pods 01:02:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:02:46 INFO [loop_until]: OK (rc = 0) 01:02:46 DEBUG --- stdout --- 01:02:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10659m 9387Mi am-6fd554f8f7-hvvkw 8436m 8876Mi am-6fd554f8f7-pmj4b 9340m 9030Mi ds-cts-0 8948m 12268Mi ds-cts-1 8526m 12259Mi ds-cts-2 8873m 12470Mi ds-idrepo-0 4839m 3257Mi ds-idrepo-1 4725m 3122Mi ds-idrepo-2 4821m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 13m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6991m 926Mi 01:02:46 DEBUG --- stderr --- 01:02:46 DEBUG 01:02:46 INFO 01:02:46 INFO [loop_until]: kubectl --namespace=xlou top node 01:02:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:02:46 INFO [loop_until]: OK (rc = 0) 01:02:46 DEBUG --- stdout --- 01:02:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5922Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4686m 15% 6667Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8461m 28% 11605Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3237Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9039m 30% 14753Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5223Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10887m 36% 12187Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27275m 91% 3118Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9970m 33% 11248Mi 10% gke-xlou-cdm-ds-0f601751-j488 4988m 16% 6082Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8643m 28% 16292Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5296m 17% 6757Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8665m 29% 14367Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 6969m 43% 4666Mi 7% 01:02:46 DEBUG --- stderr --- 01:02:46 DEBUG 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485025" 01:03:45 INFO [http_cmd]: http status code OK 01:03:45 DEBUG --- status code --- 01:03:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:45 DEBUG --- http response --- 01:03:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:03:45 DEBUG ----- output ----- 01:03:45 DEBUG { 01:03:45 DEBUG "status": "success", 01:03:45 DEBUG "data": { 01:03:45 DEBUG "resultType": "vector", 01:03:45 DEBUG "result": [ 01:03:45 DEBUG { 01:03:45 DEBUG "metric": { 01:03:45 DEBUG "instance": "10.142.0.107:9100" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG [...] 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG "instance": "10.142.0.43:9100" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 1713485025, 01:03:45 DEBUG "0.966233333333335" 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485025" 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485025" 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485025" 01:03:45 ERROR [http_cmd]: ERROR 01:03:45 ERROR --- status code --- 01:03:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:03:45 ERROR --- http response --- 01:03:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:03:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485025" 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485025" 01:03:45 INFO [http_cmd]: http status code OK 01:03:45 DEBUG --- status code --- 01:03:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:45 DEBUG --- http response --- 01:03:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:03:45 INFO [http_cmd]: http status code OK 01:03:45 DEBUG --- status code --- 01:03:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:45 DEBUG --- http response --- 01:03:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:03:45 DEBUG ----- output ----- 01:03:45 DEBUG { 01:03:45 DEBUG "status": "success", 01:03:45 DEBUG "data": { 01:03:45 DEBUG "resultType": "vector", 01:03:45 DEBUG "result": [ 01:03:45 DEBUG { 01:03:45 DEBUG "metric": { 01:03:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG [...] 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 1713485025, 01:03:45 DEBUG "532.0171450837771" 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 INFO [http_cmd]: http status code OK 01:03:45 INFO [http_cmd]: http status code OK 01:03:45 DEBUG --- status code --- 01:03:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:45 DEBUG --- http response --- 01:03:45 DEBUG --- status code --- 01:03:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:45 DEBUG --- http response --- 01:03:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:03:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:45 DEBUG ----- output ----- 01:03:45 DEBUG ----- output ----- 01:03:45 DEBUG { 01:03:45 DEBUG { 01:03:45 DEBUG "status": "success", 01:03:45 DEBUG "status": "success", 01:03:45 DEBUG "data": { 01:03:45 DEBUG "data": { 01:03:45 DEBUG "resultType": "vector", 01:03:45 DEBUG "resultType": "vector", 01:03:45 DEBUG "result": [ 01:03:45 DEBUG "result": [ 01:03:45 DEBUG { 01:03:45 DEBUG { 01:03:45 DEBUG "metric": { 01:03:45 DEBUG "metric": { 01:03:45 DEBUG "instance": "10.142.0.107:9100" 01:03:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:03:45 DEBUG }, 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG [...] 01:03:45 DEBUG [...] 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG "instance": "10.142.0.43:9100" 01:03:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:03:45 DEBUG }, 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 1713485025, 01:03:45 DEBUG 1713485025, 01:03:45 DEBUG "0" 01:03:45 DEBUG "319433464832" 01:03:45 DEBUG ] 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 DEBUG ] 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485025" 01:03:45 INFO [http_cmd]: http status code OK 01:03:45 DEBUG --- status code --- 01:03:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:45 DEBUG --- http response --- 01:03:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:03:45 DEBUG ----- output ----- 01:03:45 DEBUG { 01:03:45 DEBUG "status": "success", 01:03:45 DEBUG "data": { 01:03:45 DEBUG "resultType": "vector", 01:03:45 DEBUG "result": [ 01:03:45 DEBUG { 01:03:45 DEBUG "metric": { 01:03:45 DEBUG "pod": "ds-idrepo-0" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG [...] 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG "pod": "ds-cts-2" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 1713485025, 01:03:45 DEBUG "0" 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485025" 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485025" 01:03:45 INFO [http_cmd]: http status code OK 01:03:45 DEBUG --- status code --- 01:03:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:45 DEBUG --- http response --- 01:03:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:03:45 DEBUG ----- output ----- 01:03:45 DEBUG { 01:03:45 DEBUG "status": "success", 01:03:45 DEBUG "data": { 01:03:45 DEBUG "resultType": "vector", 01:03:45 DEBUG "result": [ 01:03:45 DEBUG { 01:03:45 DEBUG "metric": { 01:03:45 DEBUG "instance": "10.142.0.107:9100" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG [...] 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG "instance": "10.142.0.43:9100" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 1713485025, 01:03:45 DEBUG "47052029.9" 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485025" 01:03:45 INFO [http_cmd]: http status code OK 01:03:45 DEBUG --- status code --- 01:03:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:45 DEBUG --- http response --- 01:03:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:03:45 DEBUG ----- output ----- 01:03:45 DEBUG { 01:03:45 DEBUG "status": "success", 01:03:45 DEBUG "data": { 01:03:45 DEBUG "resultType": "vector", 01:03:45 DEBUG "result": [ 01:03:45 DEBUG { 01:03:45 DEBUG "metric": { 01:03:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG [...] 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 1713485025, 01:03:45 DEBUG "31389.01155994285" 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 INFO [http_cmd]: http status code OK 01:03:45 DEBUG --- status code --- 01:03:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:45 DEBUG --- http response --- 01:03:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:03:45 DEBUG ----- output ----- 01:03:45 DEBUG { 01:03:45 DEBUG "status": "success", 01:03:45 DEBUG "data": { 01:03:45 DEBUG "resultType": "vector", 01:03:45 DEBUG "result": [ 01:03:45 DEBUG { 01:03:45 DEBUG "metric": { 01:03:45 DEBUG "pod": "ds-cts-0" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG [...] 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG "pod": "ds-cts-2" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 1713485025, 01:03:45 DEBUG "0" 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485025" 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485025" 01:03:45 INFO [http_cmd]: http status code OK 01:03:45 INFO [http_cmd]: http status code OK 01:03:45 DEBUG --- status code --- 01:03:45 DEBUG --- status code --- 01:03:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:45 DEBUG --- http response --- 01:03:45 DEBUG --- http response --- 01:03:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:45 DEBUG ----- output ----- 01:03:45 DEBUG ----- output ----- 01:03:45 DEBUG { 01:03:45 DEBUG { 01:03:45 DEBUG "status": "success", 01:03:45 DEBUG "status": "success", 01:03:45 DEBUG "data": { 01:03:45 DEBUG "data": { 01:03:45 DEBUG "resultType": "vector", 01:03:45 DEBUG "resultType": "vector", 01:03:45 DEBUG "result": [ 01:03:45 DEBUG "result": [ 01:03:45 DEBUG { 01:03:45 DEBUG { 01:03:45 DEBUG "metric": { 01:03:45 DEBUG "metric": { 01:03:45 DEBUG "pod": "ds-idrepo-0" 01:03:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:03:45 DEBUG }, 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG [...] 01:03:45 DEBUG [...] 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG "pod": "ds-idrepo-2" 01:03:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:03:45 DEBUG }, 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 1713485025, 01:03:45 DEBUG 1713485025, 01:03:45 DEBUG "0" 01:03:45 DEBUG "1914.3" 01:03:45 DEBUG ] 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 DEBUG ] 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485025" 01:03:45 INFO [http_cmd]: http status code OK 01:03:45 DEBUG --- status code --- 01:03:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:45 DEBUG --- http response --- 01:03:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:03:45 DEBUG ----- output ----- 01:03:45 DEBUG { 01:03:45 DEBUG "status": "success", 01:03:45 DEBUG "data": { 01:03:45 DEBUG "resultType": "vector", 01:03:45 DEBUG "result": [ 01:03:45 DEBUG { 01:03:45 DEBUG "metric": { 01:03:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG [...] 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 1713485025, 01:03:45 DEBUG "9.215068385504606" 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485025" 01:03:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485025" 01:03:45 INFO [http_cmd]: http status code OK 01:03:45 DEBUG --- status code --- 01:03:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:45 DEBUG --- http response --- 01:03:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:45 DEBUG ----- output ----- 01:03:45 DEBUG { 01:03:45 DEBUG "status": "success", 01:03:45 DEBUG "data": { 01:03:45 DEBUG "resultType": "vector", 01:03:45 DEBUG "result": [ 01:03:45 DEBUG { 01:03:45 DEBUG "metric": { 01:03:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG [...] 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 1713485025, 01:03:45 DEBUG "0" 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:45 INFO [http_cmd]: http status code OK 01:03:45 DEBUG --- status code --- 01:03:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:45 DEBUG --- http response --- 01:03:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:03:45 DEBUG ----- output ----- 01:03:45 DEBUG { 01:03:45 DEBUG "status": "success", 01:03:45 DEBUG "data": { 01:03:45 DEBUG "resultType": "vector", 01:03:45 DEBUG "result": [ 01:03:45 DEBUG { 01:03:45 DEBUG "metric": { 01:03:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG [...] 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG 01:03:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:03:45 DEBUG }, 01:03:45 DEBUG "value": [ 01:03:45 DEBUG 1713485025, 01:03:45 DEBUG "6.89384685638354" 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG ] 01:03:45 DEBUG } 01:03:45 DEBUG } 01:03:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485026" 01:03:46 INFO [http_cmd]: http status code OK 01:03:46 DEBUG --- status code --- 01:03:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:46 DEBUG --- http response --- 01:03:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:03:46 DEBUG ----- output ----- 01:03:46 DEBUG { 01:03:46 DEBUG "status": "success", 01:03:46 DEBUG "data": { 01:03:46 DEBUG "resultType": "vector", 01:03:46 DEBUG "result": [ 01:03:46 DEBUG { 01:03:46 DEBUG "metric": { 01:03:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:03:46 DEBUG }, 01:03:46 DEBUG "value": [ 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG [...] 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:03:46 DEBUG }, 01:03:46 DEBUG "value": [ 01:03:46 DEBUG 1713485026, 01:03:46 DEBUG "0" 01:03:46 DEBUG ] 01:03:46 DEBUG } 01:03:46 DEBUG ] 01:03:46 DEBUG } 01:03:46 DEBUG } 01:03:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485026" 01:03:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485026" 01:03:46 INFO [http_cmd]: http status code OK 01:03:46 INFO [http_cmd]: http status code OK 01:03:46 DEBUG --- status code --- 01:03:46 DEBUG --- status code --- 01:03:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:46 DEBUG --- http response --- 01:03:46 DEBUG --- http response --- 01:03:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:46 DEBUG ----- output ----- 01:03:46 DEBUG ----- output ----- 01:03:46 DEBUG { 01:03:46 DEBUG { 01:03:46 DEBUG "status": "success", 01:03:46 DEBUG "status": "success", 01:03:46 DEBUG "data": { 01:03:46 DEBUG "data": { 01:03:46 DEBUG "resultType": "vector", 01:03:46 DEBUG "resultType": "vector", 01:03:46 DEBUG "result": [ 01:03:46 DEBUG "result": [ 01:03:46 DEBUG { 01:03:46 DEBUG { 01:03:46 DEBUG "metric": { 01:03:46 DEBUG "metric": { 01:03:46 DEBUG "pod": "ds-idrepo-0" 01:03:46 DEBUG "pod": "ds-idrepo-0" 01:03:46 DEBUG }, 01:03:46 DEBUG }, 01:03:46 DEBUG "value": [ 01:03:46 DEBUG "value": [ 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG [...] 01:03:46 DEBUG [...] 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG "pod": "ds-idrepo-2" 01:03:46 DEBUG "pod": "ds-idrepo-2" 01:03:46 DEBUG }, 01:03:46 DEBUG }, 01:03:46 DEBUG "value": [ 01:03:46 DEBUG "value": [ 01:03:46 DEBUG 1713485026, 01:03:46 DEBUG 1713485026, 01:03:46 DEBUG "0" 01:03:46 DEBUG "0" 01:03:46 DEBUG ] 01:03:46 DEBUG ] 01:03:46 DEBUG } 01:03:46 DEBUG } 01:03:46 DEBUG ] 01:03:46 DEBUG ] 01:03:46 DEBUG } 01:03:46 DEBUG } 01:03:46 DEBUG } 01:03:46 DEBUG } 01:03:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485026" 01:03:46 INFO [http_cmd]: http status code OK 01:03:46 DEBUG --- status code --- 01:03:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:46 DEBUG --- http response --- 01:03:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:46 DEBUG ----- output ----- 01:03:46 DEBUG { 01:03:46 DEBUG "status": "success", 01:03:46 DEBUG "data": { 01:03:46 DEBUG "resultType": "vector", 01:03:46 DEBUG "result": [ 01:03:46 DEBUG { 01:03:46 DEBUG "metric": { 01:03:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:03:46 DEBUG }, 01:03:46 DEBUG "value": [ 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG [...] 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:03:46 DEBUG }, 01:03:46 DEBUG "value": [ 01:03:46 DEBUG 1713485026, 01:03:46 DEBUG "1913.3999999999999" 01:03:46 DEBUG ] 01:03:46 DEBUG } 01:03:46 DEBUG ] 01:03:46 DEBUG } 01:03:46 DEBUG } 01:03:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485026" 01:03:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485026" 01:03:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485026" 01:03:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485026" 01:03:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485026" 01:03:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485026" 01:03:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485026" 01:03:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485026" 01:03:46 INFO [http_cmd]: http status code OK 01:03:46 INFO [http_cmd]: http status code OK 01:03:46 DEBUG --- status code --- 01:03:46 DEBUG --- status code --- 01:03:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:46 DEBUG --- http response --- 01:03:46 DEBUG --- http response --- 01:03:46 INFO [http_cmd]: http status code OK 01:03:46 INFO [http_cmd]: http status code OK 01:03:46 INFO [http_cmd]: http status code OK 01:03:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:03:46 DEBUG --- status code --- 01:03:46 DEBUG --- status code --- 01:03:46 DEBUG --- status code --- 01:03:46 DEBUG ----- output ----- 01:03:46 DEBUG ----- output ----- 01:03:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:46 DEBUG { 01:03:46 DEBUG { 01:03:46 INFO [http_cmd]: http status code OK 01:03:46 DEBUG --- http response --- 01:03:46 DEBUG --- http response --- 01:03:46 DEBUG --- http response --- 01:03:46 DEBUG "status": "success", 01:03:46 DEBUG "status": "success", 01:03:46 DEBUG --- status code --- 01:03:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:03:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:03:46 DEBUG "data": { 01:03:46 DEBUG "data": { 01:03:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:46 DEBUG ----- output ----- 01:03:46 DEBUG ----- output ----- 01:03:46 DEBUG ----- output ----- 01:03:46 DEBUG "resultType": "vector", 01:03:46 DEBUG "resultType": "vector", 01:03:46 DEBUG --- http response --- 01:03:46 DEBUG { 01:03:46 DEBUG { 01:03:46 DEBUG { 01:03:46 DEBUG "result": [ 01:03:46 DEBUG "result": [ 01:03:46 DEBUG "status": "success", 01:03:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:03:46 DEBUG "status": "success", 01:03:46 INFO [http_cmd]: http status code OK 01:03:46 DEBUG "status": "success", 01:03:46 DEBUG { 01:03:46 DEBUG "metric": { 01:03:46 DEBUG "data": { 01:03:46 DEBUG ----- output ----- 01:03:46 DEBUG "data": { 01:03:46 DEBUG --- status code --- 01:03:46 DEBUG "data": { 01:03:46 DEBUG { 01:03:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:03:46 DEBUG "resultType": "vector", 01:03:46 DEBUG { 01:03:46 DEBUG "resultType": "vector", 01:03:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:46 DEBUG "resultType": "vector", 01:03:46 DEBUG "metric": { 01:03:46 DEBUG "pod": "ds-cts-0" 01:03:46 DEBUG }, 01:03:46 DEBUG "result": [ 01:03:46 DEBUG "status": "success", 01:03:46 DEBUG "result": [ 01:03:46 DEBUG --- http response --- 01:03:46 DEBUG "result": [ 01:03:46 INFO [http_cmd]: http status code OK 01:03:46 DEBUG }, 01:03:46 DEBUG "value": [ 01:03:46 DEBUG { 01:03:46 DEBUG "data": { 01:03:46 DEBUG { 01:03:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:03:46 DEBUG { 01:03:46 DEBUG --- status code --- 01:03:46 DEBUG "value": [ 01:03:46 DEBUG 01:03:46 DEBUG "metric": { 01:03:46 DEBUG "resultType": "vector", 01:03:46 DEBUG "metric": { 01:03:46 DEBUG ----- output ----- 01:03:46 DEBUG "metric": { 01:03:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:03:46 DEBUG "result": [ 01:03:46 DEBUG "pod": "ds-cts-0" 01:03:46 DEBUG { 01:03:46 DEBUG "instance": "10.142.0.107:9100" 01:03:46 DEBUG --- http response --- 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG }, 01:03:46 DEBUG { 01:03:46 DEBUG }, 01:03:46 DEBUG "status": "success", 01:03:46 DEBUG }, 01:03:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:03:46 DEBUG 01:03:46 DEBUG [...] 01:03:46 DEBUG "value": [ 01:03:46 DEBUG "metric": { 01:03:46 DEBUG "value": [ 01:03:46 DEBUG "data": { 01:03:46 DEBUG "value": [ 01:03:46 DEBUG ----- output ----- 01:03:46 DEBUG [...] 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:03:46 DEBUG 01:03:46 DEBUG "resultType": "vector", 01:03:46 DEBUG 01:03:46 DEBUG { 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG }, 01:03:46 DEBUG "value": [ 01:03:46 DEBUG "result": [ 01:03:46 DEBUG 01:03:46 DEBUG "status": "success", 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG { 01:03:46 DEBUG 01:03:46 DEBUG "data": { 01:03:46 DEBUG 01:03:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:03:46 DEBUG [...] 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG "metric": { 01:03:46 DEBUG [...] 01:03:46 DEBUG "resultType": "vector", 01:03:46 DEBUG "pod": "ds-cts-2" 01:03:46 DEBUG }, 01:03:46 DEBUG 01:03:46 DEBUG [...] 01:03:46 DEBUG 01:03:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:03:46 DEBUG 01:03:46 DEBUG "result": [ 01:03:46 DEBUG }, 01:03:46 DEBUG "value": [ 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG [...] 01:03:46 DEBUG }, 01:03:46 DEBUG 01:03:46 DEBUG { 01:03:46 DEBUG "metric": { 01:03:46 DEBUG 1713485026, 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG "value": [ 01:03:46 DEBUG 01:03:46 DEBUG "value": [ 01:03:46 DEBUG "instance": "10.142.0.107:9100" 01:03:46 DEBUG "1914.1666666666667" 01:03:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 01:03:46 DEBUG 1713485026, 01:03:46 DEBUG }, 01:03:46 DEBUG ] 01:03:46 DEBUG }, 01:03:46 DEBUG "pod": "ds-cts-2" 01:03:46 DEBUG 01:03:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:03:46 DEBUG "instance": "10.142.0.43:9100" 01:03:46 DEBUG "0.00006666666666666667" 01:03:46 DEBUG "value": [ 01:03:46 DEBUG } 01:03:46 DEBUG "value": [ 01:03:46 DEBUG }, 01:03:46 DEBUG 01:03:46 DEBUG }, 01:03:46 DEBUG }, 01:03:46 DEBUG ] 01:03:46 DEBUG 01:03:46 DEBUG ] 01:03:46 DEBUG 1713485026, 01:03:46 DEBUG "value": [ 01:03:46 DEBUG [...] 01:03:46 DEBUG "value": [ 01:03:46 DEBUG "value": [ 01:03:46 DEBUG } 01:03:46 DEBUG 01:03:46 DEBUG } 01:03:46 DEBUG "2300152" 01:03:46 DEBUG 1713485026, 01:03:46 DEBUG 01:03:46 DEBUG 1713485026, 01:03:46 DEBUG 1713485026, 01:03:46 DEBUG ] 01:03:46 DEBUG 01:03:46 DEBUG } 01:03:46 DEBUG ] 01:03:46 DEBUG "1477.9666666666667" 01:03:46 DEBUG ] 01:03:46 DEBUG "20.076" 01:03:46 DEBUG "47056975.83333333" 01:03:46 DEBUG } 01:03:46 DEBUG } 01:03:46 DEBUG } 01:03:46 DEBUG 01:03:46 DEBUG } 01:03:46 DEBUG ] 01:03:46 DEBUG ] 01:03:46 DEBUG [...] 01:03:46 DEBUG ] 01:03:46 DEBUG 01:03:46 DEBUG ] 01:03:46 DEBUG } 01:03:46 DEBUG } 01:03:46 DEBUG 01:03:46 DEBUG } 01:03:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:03:46 DEBUG } 01:03:46 DEBUG } 01:03:46 DEBUG ] 01:03:46 DEBUG 01:03:46 DEBUG } 01:03:46 DEBUG }, 01:03:46 DEBUG ] 01:03:46 DEBUG } 01:03:46 DEBUG 01:03:46 DEBUG "value": [ 01:03:46 DEBUG } 01:03:46 DEBUG } 01:03:46 DEBUG "instance": "10.142.0.43:9100" 01:03:46 DEBUG }, 01:03:46 DEBUG } 01:03:46 DEBUG 1713485026, 01:03:46 DEBUG "value": [ 01:03:46 DEBUG "1064423424" 01:03:46 DEBUG 1713485026, 01:03:46 DEBUG ] 01:03:46 DEBUG "1.2803333333333333" 01:03:46 DEBUG } 01:03:46 DEBUG ] 01:03:46 DEBUG ] 01:03:46 DEBUG } 01:03:46 DEBUG } 01:03:46 DEBUG ] 01:03:46 DEBUG } 01:03:46 DEBUG } 01:03:46 DEBUG } 01:03:46 INFO 01:03:46 INFO [loop_until]: kubectl --namespace=xlou top pods 01:03:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:03:46 INFO [loop_until]: OK (rc = 0) 01:03:46 DEBUG --- stdout --- 01:03:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10575m 9389Mi am-6fd554f8f7-hvvkw 8508m 8875Mi am-6fd554f8f7-pmj4b 9093m 9032Mi ds-cts-0 8710m 12368Mi ds-cts-1 8389m 12271Mi ds-cts-2 8348m 12381Mi ds-idrepo-0 4763m 3257Mi ds-idrepo-1 4549m 3122Mi ds-idrepo-2 4573m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6921m 924Mi 01:03:46 DEBUG --- stderr --- 01:03:46 DEBUG 01:03:46 INFO 01:03:46 INFO [loop_until]: kubectl --namespace=xlou top node 01:03:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:03:46 INFO [loop_until]: OK (rc = 0) 01:03:46 DEBUG --- stdout --- 01:03:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5925Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4575m 15% 6672Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8311m 27% 11597Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3224Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8501m 28% 14816Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5202Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10536m 35% 12177Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27905m 93% 3135Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9949m 33% 11251Mi 10% gke-xlou-cdm-ds-0f601751-j488 4986m 16% 6079Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8226m 27% 16276Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5269m 17% 6758Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8800m 29% 14258Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7059m 44% 4664Mi 7% 01:03:46 DEBUG --- stderr --- 01:03:46 DEBUG 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485085" 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485085" 01:04:45 INFO [http_cmd]: http status code OK 01:04:45 DEBUG --- status code --- 01:04:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:45 DEBUG --- http response --- 01:04:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:04:45 DEBUG ----- output ----- 01:04:45 DEBUG { 01:04:45 DEBUG "status": "success", 01:04:45 DEBUG "data": { 01:04:45 DEBUG "resultType": "vector", 01:04:45 DEBUG "result": [ 01:04:45 DEBUG { 01:04:45 DEBUG "metric": { 01:04:45 DEBUG "instance": "10.142.0.107:9100" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG [...] 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG "instance": "10.142.0.43:9100" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 1713485085, 01:04:45 DEBUG "0.9667666666666662" 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG } 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485085" 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485085" 01:04:45 ERROR [http_cmd]: ERROR 01:04:45 ERROR --- status code --- 01:04:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:04:45 ERROR --- http response --- 01:04:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:04:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485085" 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485085" 01:04:45 INFO [http_cmd]: http status code OK 01:04:45 INFO [http_cmd]: http status code OK 01:04:45 DEBUG --- status code --- 01:04:45 DEBUG --- status code --- 01:04:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:45 DEBUG --- http response --- 01:04:45 DEBUG --- http response --- 01:04:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:04:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:04:45 DEBUG ----- output ----- 01:04:45 DEBUG { 01:04:45 DEBUG "status": "success", 01:04:45 DEBUG "data": { 01:04:45 DEBUG "resultType": "vector", 01:04:45 DEBUG "result": [ 01:04:45 DEBUG { 01:04:45 DEBUG "metric": { 01:04:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG [...] 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 1713485085, 01:04:45 DEBUG "0" 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG } 01:04:45 INFO [http_cmd]: http status code OK 01:04:45 DEBUG --- status code --- 01:04:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:45 DEBUG --- http response --- 01:04:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:04:45 DEBUG ----- output ----- 01:04:45 DEBUG { 01:04:45 DEBUG "status": "success", 01:04:45 DEBUG "data": { 01:04:45 DEBUG "resultType": "vector", 01:04:45 DEBUG "result": [ 01:04:45 DEBUG { 01:04:45 DEBUG "metric": { 01:04:45 DEBUG "instance": "10.142.0.107:9100" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG [...] 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG "instance": "10.142.0.43:9100" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 1713485085, 01:04:45 DEBUG "321844139008" 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG } 01:04:45 INFO [http_cmd]: http status code OK 01:04:45 DEBUG --- status code --- 01:04:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:45 DEBUG --- http response --- 01:04:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:45 DEBUG ----- output ----- 01:04:45 DEBUG { 01:04:45 DEBUG "status": "success", 01:04:45 DEBUG "data": { 01:04:45 DEBUG "resultType": "vector", 01:04:45 DEBUG "result": [ 01:04:45 DEBUG { 01:04:45 DEBUG "metric": { 01:04:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG [...] 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 1713485085, 01:04:45 DEBUG "0" 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG } 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485085" 01:04:45 INFO [http_cmd]: http status code OK 01:04:45 DEBUG --- status code --- 01:04:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:45 DEBUG --- http response --- 01:04:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:04:45 DEBUG ----- output ----- 01:04:45 DEBUG { 01:04:45 DEBUG "status": "success", 01:04:45 DEBUG "data": { 01:04:45 DEBUG "resultType": "vector", 01:04:45 DEBUG "result": [ 01:04:45 DEBUG { 01:04:45 DEBUG "metric": { 01:04:45 DEBUG "pod": "ds-idrepo-0" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG [...] 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG "pod": "ds-cts-2" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 1713485085, 01:04:45 DEBUG "0" 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG } 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485085" 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485085" 01:04:45 INFO [http_cmd]: http status code OK 01:04:45 DEBUG --- status code --- 01:04:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:45 DEBUG --- http response --- 01:04:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:04:45 DEBUG ----- output ----- 01:04:45 DEBUG { 01:04:45 DEBUG "status": "success", 01:04:45 DEBUG "data": { 01:04:45 DEBUG "resultType": "vector", 01:04:45 DEBUG "result": [ 01:04:45 DEBUG { 01:04:45 DEBUG "metric": { 01:04:45 DEBUG "instance": "10.142.0.107:9100" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG [...] 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG "instance": "10.142.0.43:9100" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 1713485085, 01:04:45 DEBUG "47811103.2" 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG } 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485085" 01:04:45 INFO [http_cmd]: http status code OK 01:04:45 DEBUG --- status code --- 01:04:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:45 DEBUG --- http response --- 01:04:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:04:45 DEBUG ----- output ----- 01:04:45 DEBUG { 01:04:45 DEBUG "status": "success", 01:04:45 DEBUG "data": { 01:04:45 DEBUG "resultType": "vector", 01:04:45 DEBUG "result": [ 01:04:45 DEBUG { 01:04:45 DEBUG "metric": { 01:04:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG [...] 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 1713485085, 01:04:45 DEBUG "36005.67551115898" 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG } 01:04:45 INFO [http_cmd]: http status code OK 01:04:45 DEBUG --- status code --- 01:04:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:45 DEBUG --- http response --- 01:04:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:04:45 DEBUG ----- output ----- 01:04:45 DEBUG { 01:04:45 DEBUG "status": "success", 01:04:45 DEBUG "data": { 01:04:45 DEBUG "resultType": "vector", 01:04:45 DEBUG "result": [ 01:04:45 DEBUG { 01:04:45 DEBUG "metric": { 01:04:45 DEBUG "pod": "ds-cts-0" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG [...] 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG "pod": "ds-cts-2" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 1713485085, 01:04:45 DEBUG "0" 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG } 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485085" 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485085" 01:04:45 INFO [http_cmd]: http status code OK 01:04:45 DEBUG --- status code --- 01:04:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:45 DEBUG --- http response --- 01:04:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:45 DEBUG ----- output ----- 01:04:45 DEBUG { 01:04:45 DEBUG "status": "success", 01:04:45 INFO [http_cmd]: http status code OK 01:04:45 DEBUG "data": { 01:04:45 DEBUG --- status code --- 01:04:45 DEBUG "resultType": "vector", 01:04:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:45 DEBUG "result": [ 01:04:45 DEBUG --- http response --- 01:04:45 DEBUG { 01:04:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:45 DEBUG "metric": { 01:04:45 DEBUG ----- output ----- 01:04:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:04:45 DEBUG { 01:04:45 DEBUG }, 01:04:45 DEBUG "status": "success", 01:04:45 DEBUG "value": [ 01:04:45 DEBUG "data": { 01:04:45 DEBUG 01:04:45 DEBUG "resultType": "vector", 01:04:45 DEBUG 01:04:45 DEBUG "result": [ 01:04:45 DEBUG 01:04:45 DEBUG { 01:04:45 DEBUG [...] 01:04:45 DEBUG "metric": { 01:04:45 DEBUG 01:04:45 DEBUG "pod": "ds-idrepo-0" 01:04:45 DEBUG 01:04:45 DEBUG }, 01:04:45 DEBUG 01:04:45 DEBUG "value": [ 01:04:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:04:45 DEBUG 01:04:45 DEBUG }, 01:04:45 DEBUG 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 01:04:45 DEBUG 1713485085, 01:04:45 DEBUG [...] 01:04:45 DEBUG "1926.9" 01:04:45 DEBUG 01:04:45 DEBUG ] 01:04:45 DEBUG 01:04:45 DEBUG } 01:04:45 DEBUG 01:04:45 DEBUG ] 01:04:45 DEBUG "pod": "ds-idrepo-2" 01:04:45 DEBUG } 01:04:45 DEBUG }, 01:04:45 DEBUG } 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 1713485085, 01:04:45 DEBUG "0" 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG } 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485085" 01:04:45 INFO [http_cmd]: http status code OK 01:04:45 DEBUG --- status code --- 01:04:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:45 DEBUG --- http response --- 01:04:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:04:45 DEBUG ----- output ----- 01:04:45 DEBUG { 01:04:45 DEBUG "status": "success", 01:04:45 DEBUG "data": { 01:04:45 DEBUG "resultType": "vector", 01:04:45 DEBUG "result": [ 01:04:45 DEBUG { 01:04:45 DEBUG "metric": { 01:04:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG [...] 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 1713485085, 01:04:45 DEBUG "13.773100608124741" 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG } 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485085" 01:04:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485085" 01:04:45 INFO [http_cmd]: http status code OK 01:04:45 DEBUG --- status code --- 01:04:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:45 DEBUG --- http response --- 01:04:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:45 DEBUG ----- output ----- 01:04:45 DEBUG { 01:04:45 DEBUG "status": "success", 01:04:45 DEBUG "data": { 01:04:45 DEBUG "resultType": "vector", 01:04:45 DEBUG "result": [ 01:04:45 DEBUG { 01:04:45 DEBUG "metric": { 01:04:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG [...] 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 1713485085, 01:04:45 DEBUG "0" 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG } 01:04:45 INFO [http_cmd]: http status code OK 01:04:45 DEBUG --- status code --- 01:04:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:45 DEBUG --- http response --- 01:04:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:04:45 DEBUG ----- output ----- 01:04:45 DEBUG { 01:04:45 DEBUG "status": "success", 01:04:45 DEBUG "data": { 01:04:45 DEBUG "resultType": "vector", 01:04:45 DEBUG "result": [ 01:04:45 DEBUG { 01:04:45 DEBUG "metric": { 01:04:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG [...] 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG 01:04:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:04:45 DEBUG }, 01:04:45 DEBUG "value": [ 01:04:45 DEBUG 1713485085, 01:04:45 DEBUG "6.912611576484861" 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG ] 01:04:45 DEBUG } 01:04:45 DEBUG } 01:04:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485086" 01:04:46 INFO [http_cmd]: http status code OK 01:04:46 DEBUG --- status code --- 01:04:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:46 DEBUG --- http response --- 01:04:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:04:46 DEBUG ----- output ----- 01:04:46 DEBUG { 01:04:46 DEBUG "status": "success", 01:04:46 DEBUG "data": { 01:04:46 DEBUG "resultType": "vector", 01:04:46 DEBUG "result": [ 01:04:46 DEBUG { 01:04:46 DEBUG "metric": { 01:04:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:04:46 DEBUG }, 01:04:46 DEBUG "value": [ 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG [...] 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:04:46 DEBUG }, 01:04:46 DEBUG "value": [ 01:04:46 DEBUG 1713485086, 01:04:46 DEBUG "0" 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG } 01:04:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485086" 01:04:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485086" 01:04:46 INFO [http_cmd]: http status code OK 01:04:46 DEBUG --- status code --- 01:04:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:46 DEBUG --- http response --- 01:04:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:46 DEBUG ----- output ----- 01:04:46 DEBUG { 01:04:46 DEBUG "status": "success", 01:04:46 DEBUG "data": { 01:04:46 DEBUG "resultType": "vector", 01:04:46 DEBUG "result": [ 01:04:46 DEBUG { 01:04:46 DEBUG "metric": { 01:04:46 DEBUG "pod": "ds-idrepo-0" 01:04:46 DEBUG }, 01:04:46 DEBUG "value": [ 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG [...] 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG "pod": "ds-idrepo-2" 01:04:46 DEBUG }, 01:04:46 DEBUG "value": [ 01:04:46 DEBUG 1713485086, 01:04:46 DEBUG "0" 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG } 01:04:46 INFO [http_cmd]: http status code OK 01:04:46 DEBUG --- status code --- 01:04:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:46 DEBUG --- http response --- 01:04:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:46 DEBUG ----- output ----- 01:04:46 DEBUG { 01:04:46 DEBUG "status": "success", 01:04:46 DEBUG "data": { 01:04:46 DEBUG "resultType": "vector", 01:04:46 DEBUG "result": [ 01:04:46 DEBUG { 01:04:46 DEBUG "metric": { 01:04:46 DEBUG "pod": "ds-idrepo-0" 01:04:46 DEBUG }, 01:04:46 DEBUG "value": [ 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG [...] 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG "pod": "ds-idrepo-2" 01:04:46 DEBUG }, 01:04:46 DEBUG "value": [ 01:04:46 DEBUG 1713485086, 01:04:46 DEBUG "0" 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG } 01:04:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485086" 01:04:46 INFO [http_cmd]: http status code OK 01:04:46 DEBUG --- status code --- 01:04:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:46 DEBUG --- http response --- 01:04:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:46 DEBUG ----- output ----- 01:04:46 DEBUG { 01:04:46 DEBUG "status": "success", 01:04:46 DEBUG "data": { 01:04:46 DEBUG "resultType": "vector", 01:04:46 DEBUG "result": [ 01:04:46 DEBUG { 01:04:46 DEBUG "metric": { 01:04:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:04:46 DEBUG }, 01:04:46 DEBUG "value": [ 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG [...] 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:04:46 DEBUG }, 01:04:46 DEBUG "value": [ 01:04:46 DEBUG 1713485086, 01:04:46 DEBUG "1925.8999999999999" 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG } 01:04:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485086" 01:04:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485086" 01:04:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485086" 01:04:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485086" 01:04:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485086" 01:04:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485086" 01:04:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485086" 01:04:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485086" 01:04:46 INFO [http_cmd]: http status code OK 01:04:46 INFO [http_cmd]: http status code OK 01:04:46 DEBUG --- status code --- 01:04:46 INFO [http_cmd]: http status code OK 01:04:46 INFO [http_cmd]: http status code OK 01:04:46 DEBUG --- status code --- 01:04:46 INFO [http_cmd]: http status code OK 01:04:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:46 DEBUG --- status code --- 01:04:46 DEBUG --- status code --- 01:04:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:46 DEBUG --- status code --- 01:04:46 DEBUG --- http response --- 01:04:46 INFO [http_cmd]: http status code OK 01:04:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:46 DEBUG --- http response --- 01:04:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:46 DEBUG --- status code --- 01:04:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:46 DEBUG --- http response --- 01:04:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:04:46 DEBUG --- http response --- 01:04:46 DEBUG ----- output ----- 01:04:46 DEBUG --- http response --- 01:04:46 DEBUG --- http response --- 01:04:46 DEBUG ----- output ----- 01:04:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:04:46 DEBUG { 01:04:46 INFO [http_cmd]: http status code OK 01:04:46 INFO [http_cmd]: http status code OK 01:04:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:04:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:04:46 DEBUG { 01:04:46 DEBUG ----- output ----- 01:04:46 DEBUG ----- output ----- 01:04:46 DEBUG "status": "success", 01:04:46 DEBUG --- status code --- 01:04:46 DEBUG --- status code --- 01:04:46 DEBUG ----- output ----- 01:04:46 DEBUG ----- output ----- 01:04:46 DEBUG "status": "success", 01:04:46 DEBUG { 01:04:46 DEBUG { 01:04:46 DEBUG "data": { 01:04:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:04:46 DEBUG { 01:04:46 DEBUG { 01:04:46 DEBUG "data": { 01:04:46 DEBUG "status": "success", 01:04:46 DEBUG "status": "success", 01:04:46 DEBUG "resultType": "vector", 01:04:46 DEBUG --- http response --- 01:04:46 DEBUG --- http response --- 01:04:46 DEBUG "status": "success", 01:04:46 DEBUG "status": "success", 01:04:46 DEBUG "resultType": "vector", 01:04:46 DEBUG "data": { 01:04:46 DEBUG "data": { 01:04:46 DEBUG "result": [ 01:04:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:04:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:04:46 DEBUG "data": { 01:04:46 DEBUG "data": { 01:04:46 DEBUG "result": [ 01:04:46 DEBUG "resultType": "vector", 01:04:46 DEBUG "resultType": "vector", 01:04:46 DEBUG { 01:04:46 DEBUG ----- output ----- 01:04:46 DEBUG ----- output ----- 01:04:46 DEBUG "resultType": "vector", 01:04:46 DEBUG "resultType": "vector", 01:04:46 DEBUG { 01:04:46 DEBUG "result": [ 01:04:46 DEBUG "result": [ 01:04:46 DEBUG "metric": { 01:04:46 DEBUG { 01:04:46 DEBUG { 01:04:46 DEBUG "result": [ 01:04:46 DEBUG "result": [ 01:04:46 DEBUG "metric": { 01:04:46 DEBUG { 01:04:46 DEBUG { 01:04:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:04:46 DEBUG }, 01:04:46 DEBUG "status": "success", 01:04:46 DEBUG { 01:04:46 DEBUG { 01:04:46 DEBUG "pod": "ds-cts-0" 01:04:46 DEBUG "metric": { 01:04:46 DEBUG "metric": { 01:04:46 DEBUG "value": [ 01:04:46 DEBUG "data": { 01:04:46 DEBUG "metric": { 01:04:46 DEBUG "metric": { 01:04:46 DEBUG "status": "success", 01:04:46 DEBUG }, 01:04:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:04:46 DEBUG "instance": "10.142.0.107:9100" 01:04:46 DEBUG 01:04:46 DEBUG "resultType": "vector", 01:04:46 DEBUG "pod": "ds-cts-0" 01:04:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:04:46 DEBUG "data": { 01:04:46 DEBUG "value": [ 01:04:46 DEBUG }, 01:04:46 DEBUG }, 01:04:46 DEBUG 01:04:46 DEBUG "result": [ 01:04:46 DEBUG }, 01:04:46 DEBUG }, 01:04:46 DEBUG "resultType": "vector", 01:04:46 DEBUG 01:04:46 DEBUG "value": [ 01:04:46 DEBUG "value": [ 01:04:46 DEBUG 01:04:46 DEBUG { 01:04:46 DEBUG "value": [ 01:04:46 DEBUG "value": [ 01:04:46 DEBUG "result": [ 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG [...] 01:04:46 DEBUG "metric": { 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG { 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG "metric": { 01:04:46 DEBUG [...] 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG }, 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG "instance": "10.142.0.107:9100" 01:04:46 DEBUG }, 01:04:46 DEBUG [...] 01:04:46 DEBUG [...] 01:04:46 DEBUG 01:04:46 DEBUG "value": [ 01:04:46 DEBUG [...] 01:04:46 DEBUG [...] 01:04:46 DEBUG 01:04:46 DEBUG "value": [ 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG }, 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG "pod": "ds-cts-2" 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG "value": [ 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG 01:04:46 DEBUG }, 01:04:46 DEBUG 01:04:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:04:46 DEBUG "instance": "10.142.0.43:9100" 01:04:46 DEBUG 1713485086, 01:04:46 DEBUG [...] 01:04:46 DEBUG 01:04:46 DEBUG "pod": "ds-cts-2" 01:04:46 DEBUG }, 01:04:46 DEBUG [...] 01:04:46 DEBUG }, 01:04:46 DEBUG }, 01:04:46 DEBUG "1926.7" 01:04:46 DEBUG 01:04:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:04:46 DEBUG "value": [ 01:04:46 DEBUG "value": [ 01:04:46 DEBUG 01:04:46 DEBUG "value": [ 01:04:46 DEBUG "value": [ 01:04:46 DEBUG ] 01:04:46 DEBUG 01:04:46 DEBUG }, 01:04:46 DEBUG 1713485086, 01:04:46 DEBUG 1713485086, 01:04:46 DEBUG 01:04:46 DEBUG 1713485086, 01:04:46 DEBUG 1713485086, 01:04:46 DEBUG } 01:04:46 DEBUG 01:04:46 DEBUG "value": [ 01:04:46 DEBUG "2917.8" 01:04:46 DEBUG "0" 01:04:46 DEBUG 01:04:46 DEBUG "2415160" 01:04:46 DEBUG "47815940.36666666" 01:04:46 DEBUG ] 01:04:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:04:46 DEBUG 1713485086, 01:04:46 DEBUG ] 01:04:46 DEBUG ] 01:04:46 DEBUG "instance": "10.142.0.43:9100" 01:04:46 DEBUG ] 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG }, 01:04:46 DEBUG "20.076" 01:04:46 DEBUG } 01:04:46 DEBUG } 01:04:46 DEBUG ] 01:04:46 DEBUG }, 01:04:46 DEBUG } 01:04:46 DEBUG } 01:04:46 DEBUG "value": [ 01:04:46 DEBUG 1713485086, 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG } 01:04:46 DEBUG "value": [ 01:04:46 DEBUG ] 01:04:46 DEBUG ] 01:04:46 DEBUG "1066782720" 01:04:46 DEBUG } 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG 1713485086, 01:04:46 DEBUG } 01:04:46 DEBUG } 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG } 01:04:46 DEBUG "1.2309999999999999" 01:04:46 DEBUG } 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG } 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG } 01:04:46 DEBUG } 01:04:46 DEBUG ] 01:04:46 DEBUG } 01:04:46 DEBUG } 01:04:46 DEBUG } 01:04:46 INFO 01:04:46 INFO [loop_until]: kubectl --namespace=xlou top pods 01:04:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:04:46 INFO [loop_until]: OK (rc = 0) 01:04:46 DEBUG --- stdout --- 01:04:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10420m 9391Mi am-6fd554f8f7-hvvkw 8749m 8876Mi am-6fd554f8f7-pmj4b 9461m 9032Mi ds-cts-0 8483m 12304Mi ds-cts-1 8551m 12382Mi ds-cts-2 9894m 12378Mi ds-idrepo-0 4618m 3257Mi ds-idrepo-1 4633m 3122Mi ds-idrepo-2 4551m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6862m 926Mi 01:04:46 DEBUG --- stderr --- 01:04:46 DEBUG 01:04:46 INFO 01:04:46 INFO [loop_until]: kubectl --namespace=xlou top node 01:04:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:04:46 INFO [loop_until]: OK (rc = 0) 01:04:46 DEBUG --- stdout --- 01:04:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5913Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4635m 15% 6656Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8446m 28% 11608Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3228Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8573m 28% 14771Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5205Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10431m 34% 12189Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28047m 93% 3125Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10005m 33% 11262Mi 10% gke-xlou-cdm-ds-0f601751-j488 5015m 16% 6087Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8905m 29% 16396Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5249m 17% 6765Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8878m 29% 14329Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7068m 44% 4664Mi 7% 01:04:46 DEBUG --- stderr --- 01:04:46 DEBUG 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485145" 01:05:45 INFO [http_cmd]: http status code OK 01:05:45 DEBUG --- status code --- 01:05:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:45 DEBUG --- http response --- 01:05:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:45 DEBUG ----- output ----- 01:05:45 DEBUG { 01:05:45 DEBUG "status": "success", 01:05:45 DEBUG "data": { 01:05:45 DEBUG "resultType": "vector", 01:05:45 DEBUG "result": [ 01:05:45 DEBUG { 01:05:45 DEBUG "metric": { 01:05:45 DEBUG "instance": "10.142.0.107:9100" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG [...] 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG "instance": "10.142.0.43:9100" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 1713485145, 01:05:45 DEBUG "0.952533333333335" 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG } 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485145" 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485145" 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485145" 01:05:45 ERROR [http_cmd]: ERROR 01:05:45 ERROR --- status code --- 01:05:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:05:45 ERROR --- http response --- 01:05:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:05:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485145" 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485145" 01:05:45 INFO [http_cmd]: http status code OK 01:05:45 INFO [http_cmd]: http status code OK 01:05:45 DEBUG --- status code --- 01:05:45 DEBUG --- status code --- 01:05:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:45 DEBUG --- http response --- 01:05:45 DEBUG --- http response --- 01:05:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:05:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:05:45 INFO [http_cmd]: http status code OK 01:05:45 DEBUG ----- output ----- 01:05:45 DEBUG --- status code --- 01:05:45 DEBUG { 01:05:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:45 DEBUG "status": "success", 01:05:45 DEBUG --- http response --- 01:05:45 DEBUG "data": { 01:05:45 DEBUG "resultType": "vector", 01:05:45 DEBUG "result": [ 01:05:45 DEBUG { 01:05:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:45 DEBUG "metric": { 01:05:45 DEBUG ----- output ----- 01:05:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:05:45 DEBUG { 01:05:45 DEBUG }, 01:05:45 DEBUG "status": "success", 01:05:45 DEBUG "value": [ 01:05:45 DEBUG "data": { 01:05:45 DEBUG 01:05:45 DEBUG "resultType": "vector", 01:05:45 DEBUG 01:05:45 DEBUG "result": [ 01:05:45 DEBUG 01:05:45 DEBUG { 01:05:45 DEBUG [...] 01:05:45 DEBUG "metric": { 01:05:45 DEBUG 01:05:45 DEBUG "instance": "10.142.0.107:9100" 01:05:45 DEBUG 01:05:45 DEBUG }, 01:05:45 DEBUG 01:05:45 DEBUG "value": [ 01:05:45 DEBUG "pod": "idm-b85f9d978-wnhxb" 01:05:45 DEBUG 01:05:45 DEBUG }, 01:05:45 DEBUG 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 01:05:45 DEBUG 1713485145, 01:05:45 DEBUG [...] 01:05:45 DEBUG "311.6295269756078" 01:05:45 DEBUG 01:05:45 DEBUG ] 01:05:45 DEBUG 01:05:45 DEBUG } 01:05:45 DEBUG 01:05:45 DEBUG ] 01:05:45 DEBUG "instance": "10.142.0.43:9100" 01:05:45 DEBUG } 01:05:45 DEBUG }, 01:05:45 DEBUG } 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 1713485145, 01:05:45 DEBUG "324243454976" 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG } 01:05:45 INFO [http_cmd]: http status code OK 01:05:45 DEBUG --- status code --- 01:05:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:45 DEBUG --- http response --- 01:05:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:45 DEBUG ----- output ----- 01:05:45 DEBUG { 01:05:45 DEBUG "status": "success", 01:05:45 DEBUG "data": { 01:05:45 DEBUG "resultType": "vector", 01:05:45 DEBUG "result": [ 01:05:45 DEBUG { 01:05:45 DEBUG "metric": { 01:05:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG [...] 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 1713485145, 01:05:45 DEBUG "0" 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG } 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485145" 01:05:45 INFO [http_cmd]: http status code OK 01:05:45 DEBUG --- status code --- 01:05:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:45 DEBUG --- http response --- 01:05:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:05:45 DEBUG ----- output ----- 01:05:45 DEBUG { 01:05:45 DEBUG "status": "success", 01:05:45 DEBUG "data": { 01:05:45 DEBUG "resultType": "vector", 01:05:45 DEBUG "result": [ 01:05:45 DEBUG { 01:05:45 DEBUG "metric": { 01:05:45 DEBUG "pod": "ds-idrepo-0" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG [...] 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG "pod": "ds-cts-2" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 1713485145, 01:05:45 DEBUG "0" 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG } 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485145" 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485145" 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485145" 01:05:45 INFO [http_cmd]: http status code OK 01:05:45 DEBUG --- status code --- 01:05:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:45 DEBUG --- http response --- 01:05:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:45 DEBUG ----- output ----- 01:05:45 DEBUG { 01:05:45 DEBUG "status": "success", 01:05:45 DEBUG "data": { 01:05:45 DEBUG "resultType": "vector", 01:05:45 DEBUG "result": [ 01:05:45 DEBUG { 01:05:45 DEBUG "metric": { 01:05:45 DEBUG "instance": "10.142.0.107:9100" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG [...] 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG "instance": "10.142.0.43:9100" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 1713485145, 01:05:45 DEBUG "47255599.16666667" 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG } 01:05:45 INFO [http_cmd]: http status code OK 01:05:45 DEBUG --- status code --- 01:05:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:45 DEBUG --- http response --- 01:05:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:05:45 DEBUG ----- output ----- 01:05:45 DEBUG { 01:05:45 DEBUG "status": "success", 01:05:45 DEBUG "data": { 01:05:45 DEBUG "resultType": "vector", 01:05:45 DEBUG "result": [ 01:05:45 DEBUG { 01:05:45 DEBUG "metric": { 01:05:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG [...] 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG "pod": "idm-b85f9d978-wnhxb" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 1713485145, 01:05:45 DEBUG "623.2590539512156" 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG } 01:05:45 INFO [http_cmd]: http status code OK 01:05:45 DEBUG --- status code --- 01:05:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:45 DEBUG --- http response --- 01:05:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:05:45 DEBUG ----- output ----- 01:05:45 DEBUG { 01:05:45 DEBUG "status": "success", 01:05:45 DEBUG "data": { 01:05:45 DEBUG "resultType": "vector", 01:05:45 DEBUG "result": [ 01:05:45 DEBUG { 01:05:45 DEBUG "metric": { 01:05:45 DEBUG "pod": "ds-cts-0" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG [...] 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG "pod": "ds-cts-2" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 1713485145, 01:05:45 DEBUG "0" 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG } 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485145" 01:05:45 INFO [http_cmd]: http status code OK 01:05:45 DEBUG --- status code --- 01:05:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:45 DEBUG --- http response --- 01:05:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:45 DEBUG ----- output ----- 01:05:45 DEBUG { 01:05:45 DEBUG "status": "success", 01:05:45 DEBUG "data": { 01:05:45 DEBUG "resultType": "vector", 01:05:45 DEBUG "result": [ 01:05:45 DEBUG { 01:05:45 DEBUG "metric": { 01:05:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG [...] 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 1713485145, 01:05:45 DEBUG "1915.1" 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG } 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485145" 01:05:45 INFO [http_cmd]: http status code OK 01:05:45 DEBUG --- status code --- 01:05:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:45 DEBUG --- http response --- 01:05:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:45 DEBUG ----- output ----- 01:05:45 DEBUG { 01:05:45 DEBUG "status": "success", 01:05:45 DEBUG "data": { 01:05:45 DEBUG "resultType": "vector", 01:05:45 DEBUG "result": [ 01:05:45 DEBUG { 01:05:45 DEBUG "metric": { 01:05:45 DEBUG "pod": "ds-idrepo-0" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG [...] 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG "pod": "ds-idrepo-2" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 1713485145, 01:05:45 DEBUG "0" 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG } 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485145" 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485145" 01:05:45 INFO [http_cmd]: http status code OK 01:05:45 DEBUG --- status code --- 01:05:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:45 DEBUG --- http response --- 01:05:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:05:45 INFO [http_cmd]: http status code OK 01:05:45 DEBUG ----- output ----- 01:05:45 DEBUG --- status code --- 01:05:45 DEBUG { 01:05:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:45 DEBUG "status": "success", 01:05:45 DEBUG --- http response --- 01:05:45 DEBUG "data": { 01:05:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:45 DEBUG "resultType": "vector", 01:05:45 DEBUG ----- output ----- 01:05:45 DEBUG "result": [ 01:05:45 DEBUG { 01:05:45 DEBUG { 01:05:45 DEBUG "status": "success", 01:05:45 DEBUG "metric": { 01:05:45 DEBUG "data": { 01:05:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:05:45 DEBUG "resultType": "vector", 01:05:45 DEBUG }, 01:05:45 DEBUG "result": [ 01:05:45 DEBUG "value": [ 01:05:45 DEBUG { 01:05:45 DEBUG 01:05:45 DEBUG "metric": { 01:05:45 DEBUG 01:05:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:05:45 DEBUG 01:05:45 DEBUG }, 01:05:45 DEBUG [...] 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG "pod": "idm-b85f9d978-wnhxb" 01:05:45 DEBUG [...] 01:05:45 DEBUG }, 01:05:45 DEBUG 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 01:05:45 DEBUG 1713485145, 01:05:45 DEBUG 01:05:45 DEBUG "0.0019016806404459961" 01:05:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:05:45 DEBUG ] 01:05:45 DEBUG }, 01:05:45 DEBUG } 01:05:45 DEBUG "value": [ 01:05:45 DEBUG ] 01:05:45 DEBUG 1713485145, 01:05:45 DEBUG } 01:05:45 DEBUG "6.90066562809456" 01:05:45 DEBUG } 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG } 01:05:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485145" 01:05:45 INFO [http_cmd]: http status code OK 01:05:45 DEBUG --- status code --- 01:05:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:45 DEBUG --- http response --- 01:05:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:45 DEBUG ----- output ----- 01:05:45 DEBUG { 01:05:45 DEBUG "status": "success", 01:05:45 DEBUG "data": { 01:05:45 DEBUG "resultType": "vector", 01:05:45 DEBUG "result": [ 01:05:45 DEBUG { 01:05:45 DEBUG "metric": { 01:05:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG [...] 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG 01:05:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:05:45 DEBUG }, 01:05:45 DEBUG "value": [ 01:05:45 DEBUG 1713485145, 01:05:45 DEBUG "0" 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG ] 01:05:45 DEBUG } 01:05:45 DEBUG } 01:05:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485146" 01:05:46 INFO [http_cmd]: http status code OK 01:05:46 DEBUG --- status code --- 01:05:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:46 DEBUG --- http response --- 01:05:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:46 DEBUG ----- output ----- 01:05:46 DEBUG { 01:05:46 DEBUG "status": "success", 01:05:46 DEBUG "data": { 01:05:46 DEBUG "resultType": "vector", 01:05:46 DEBUG "result": [ 01:05:46 DEBUG { 01:05:46 DEBUG "metric": { 01:05:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:05:46 DEBUG }, 01:05:46 DEBUG "value": [ 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG [...] 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG "pod": "ds-cts-2" 01:05:46 DEBUG }, 01:05:46 DEBUG "value": [ 01:05:46 DEBUG 1713485146, 01:05:46 DEBUG "11188653.928079488" 01:05:46 DEBUG ] 01:05:46 DEBUG } 01:05:46 DEBUG ] 01:05:46 DEBUG } 01:05:46 DEBUG } 01:05:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485146" 01:05:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485146" 01:05:46 INFO [http_cmd]: http status code OK 01:05:46 INFO [http_cmd]: http status code OK 01:05:46 DEBUG --- status code --- 01:05:46 DEBUG --- status code --- 01:05:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:46 DEBUG --- http response --- 01:05:46 DEBUG --- http response --- 01:05:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:46 DEBUG ----- output ----- 01:05:46 DEBUG { 01:05:46 DEBUG "status": "success", 01:05:46 DEBUG "data": { 01:05:46 DEBUG "resultType": "vector", 01:05:46 DEBUG "result": [ 01:05:46 DEBUG { 01:05:46 DEBUG "metric": { 01:05:46 DEBUG "pod": "ds-idrepo-0" 01:05:46 DEBUG }, 01:05:46 DEBUG "value": [ 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG [...] 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:46 DEBUG "pod": "ds-idrepo-2" 01:05:46 DEBUG ----- output ----- 01:05:46 DEBUG }, 01:05:46 DEBUG { 01:05:46 DEBUG "value": [ 01:05:46 DEBUG "status": "success", 01:05:46 DEBUG 1713485146, 01:05:46 DEBUG "data": { 01:05:46 DEBUG "0" 01:05:46 DEBUG "resultType": "vector", 01:05:46 DEBUG ] 01:05:46 DEBUG "result": [ 01:05:46 DEBUG } 01:05:46 DEBUG { 01:05:46 DEBUG ] 01:05:46 DEBUG "metric": { 01:05:46 DEBUG } 01:05:46 DEBUG "pod": "ds-idrepo-0" 01:05:46 DEBUG } 01:05:46 DEBUG }, 01:05:46 DEBUG "value": [ 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG [...] 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG "pod": "ds-idrepo-2" 01:05:46 DEBUG }, 01:05:46 DEBUG "value": [ 01:05:46 DEBUG 1713485146, 01:05:46 DEBUG "0" 01:05:46 DEBUG ] 01:05:46 DEBUG } 01:05:46 DEBUG ] 01:05:46 DEBUG } 01:05:46 DEBUG } 01:05:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485146" 01:05:46 INFO [http_cmd]: http status code OK 01:05:46 DEBUG --- status code --- 01:05:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:46 DEBUG --- http response --- 01:05:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:46 DEBUG ----- output ----- 01:05:46 DEBUG { 01:05:46 DEBUG "status": "success", 01:05:46 DEBUG "data": { 01:05:46 DEBUG "resultType": "vector", 01:05:46 DEBUG "result": [ 01:05:46 DEBUG { 01:05:46 DEBUG "metric": { 01:05:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:05:46 DEBUG }, 01:05:46 DEBUG "value": [ 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG [...] 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:05:46 DEBUG }, 01:05:46 DEBUG "value": [ 01:05:46 DEBUG 1713485146, 01:05:46 DEBUG "1915.0333333333333" 01:05:46 DEBUG ] 01:05:46 DEBUG } 01:05:46 DEBUG ] 01:05:46 DEBUG } 01:05:46 DEBUG } 01:05:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485146" 01:05:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485146" 01:05:46 INFO [http_cmd]: http status code OK 01:05:46 INFO [http_cmd]: http status code OK 01:05:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485146" 01:05:46 DEBUG --- status code --- 01:05:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:46 DEBUG --- http response --- 01:05:46 DEBUG --- status code --- 01:05:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485146" 01:05:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485146" 01:05:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485146" 01:05:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485146" 01:05:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485146" 01:05:46 DEBUG --- http response --- 01:05:46 DEBUG ----- output ----- 01:05:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:46 DEBUG { 01:05:46 DEBUG ----- output ----- 01:05:46 DEBUG "status": "success", 01:05:46 DEBUG { 01:05:46 DEBUG "data": { 01:05:46 DEBUG "status": "success", 01:05:46 DEBUG "resultType": "vector", 01:05:46 DEBUG "data": { 01:05:46 DEBUG "result": [ 01:05:46 DEBUG "resultType": "vector", 01:05:46 DEBUG { 01:05:46 DEBUG "result": [ 01:05:46 DEBUG "metric": { 01:05:46 DEBUG { 01:05:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:05:46 DEBUG "metric": { 01:05:46 INFO [http_cmd]: http status code OK 01:05:46 DEBUG }, 01:05:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:05:46 DEBUG --- status code --- 01:05:46 DEBUG "value": [ 01:05:46 DEBUG }, 01:05:46 INFO [http_cmd]: http status code OK 01:05:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:46 INFO [http_cmd]: http status code OK 01:05:46 DEBUG 01:05:46 DEBUG "value": [ 01:05:46 DEBUG --- status code --- 01:05:46 DEBUG --- http response --- 01:05:46 DEBUG --- status code --- 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:05:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG --- http response --- 01:05:46 DEBUG ----- output ----- 01:05:46 DEBUG --- http response --- 01:05:46 DEBUG [...] 01:05:46 DEBUG 01:05:46 DEBUG { 01:05:46 DEBUG "status": "success", 01:05:46 DEBUG 01:05:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:05:46 DEBUG [...] 01:05:46 INFO [http_cmd]: http status code OK 01:05:46 INFO [http_cmd]: http status code OK 01:05:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:05:46 DEBUG "data": { 01:05:46 DEBUG 01:05:46 DEBUG ----- output ----- 01:05:46 DEBUG 01:05:46 DEBUG --- status code --- 01:05:46 DEBUG --- status code --- 01:05:46 DEBUG ----- output ----- 01:05:46 DEBUG "resultType": "vector", 01:05:46 DEBUG 01:05:46 DEBUG { 01:05:46 DEBUG 01:05:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:46 DEBUG { 01:05:46 DEBUG "result": [ 01:05:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:05:46 DEBUG "status": "success", 01:05:46 DEBUG 01:05:46 DEBUG --- http response --- 01:05:46 DEBUG --- http response --- 01:05:46 DEBUG "status": "success", 01:05:46 DEBUG { 01:05:46 DEBUG }, 01:05:46 DEBUG "data": { 01:05:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:05:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:46 DEBUG "data": { 01:05:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:46 DEBUG "metric": { 01:05:46 INFO [http_cmd]: http status code OK 01:05:46 DEBUG "value": [ 01:05:46 DEBUG "resultType": "vector", 01:05:46 DEBUG }, 01:05:46 DEBUG ----- output ----- 01:05:46 DEBUG "resultType": "vector", 01:05:46 DEBUG ----- output ----- 01:05:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:05:46 DEBUG --- status code --- 01:05:46 DEBUG 1713485146, 01:05:46 DEBUG "result": [ 01:05:46 DEBUG "value": [ 01:05:46 DEBUG { 01:05:46 DEBUG "result": [ 01:05:46 DEBUG { 01:05:46 DEBUG }, 01:05:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:05:46 DEBUG "20.076" 01:05:46 DEBUG ] 01:05:46 DEBUG } 01:05:46 DEBUG { 01:05:46 DEBUG { 01:05:46 DEBUG "status": "success", 01:05:46 DEBUG "value": [ 01:05:46 DEBUG --- http response --- 01:05:46 DEBUG 1713485146, 01:05:46 DEBUG "status": "success", 01:05:46 DEBUG ] 01:05:46 DEBUG "metric": { 01:05:46 DEBUG "metric": { 01:05:46 DEBUG "data": { 01:05:46 DEBUG 01:05:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:05:46 DEBUG "2530171" 01:05:46 DEBUG "data": { 01:05:46 DEBUG } 01:05:46 DEBUG "pod": "ds-cts-0" 01:05:46 DEBUG }, 01:05:46 DEBUG "resultType": "vector", 01:05:46 DEBUG 01:05:46 DEBUG ----- output ----- 01:05:46 DEBUG ] 01:05:46 DEBUG "resultType": "vector", 01:05:46 DEBUG } 01:05:46 DEBUG "pod": "ds-cts-0" 01:05:46 DEBUG "value": [ 01:05:46 DEBUG "result": [ 01:05:46 DEBUG 01:05:46 DEBUG { 01:05:46 DEBUG } 01:05:46 DEBUG "result": [ 01:05:46 DEBUG }, 01:05:46 DEBUG 01:05:46 DEBUG { 01:05:46 DEBUG [...] 01:05:46 DEBUG "status": "success", 01:05:46 DEBUG ] 01:05:46 DEBUG { 01:05:46 DEBUG "value": [ 01:05:46 DEBUG 01:05:46 DEBUG "metric": { 01:05:46 DEBUG 01:05:46 DEBUG "data": { 01:05:46 DEBUG } 01:05:46 DEBUG "metric": { 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:05:46 DEBUG 01:05:46 DEBUG "resultType": "vector", 01:05:46 DEBUG } 01:05:46 DEBUG "instance": "10.142.0.107:9100" 01:05:46 DEBUG 01:05:46 DEBUG [...] 01:05:46 DEBUG }, 01:05:46 DEBUG 01:05:46 DEBUG "result": [ 01:05:46 DEBUG }, 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG "value": [ 01:05:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:05:46 DEBUG { 01:05:46 DEBUG "value": [ 01:05:46 DEBUG [...] 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG "metric": { 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG }, 01:05:46 DEBUG 01:05:46 DEBUG "instance": "10.142.0.107:9100" 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG "pod": "ds-cts-2" 01:05:46 DEBUG "value": [ 01:05:46 DEBUG [...] 01:05:46 DEBUG }, 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG }, 01:05:46 DEBUG 1713485146, 01:05:46 DEBUG 01:05:46 DEBUG "value": [ 01:05:46 DEBUG [...] 01:05:46 DEBUG "pod": "ds-cts-2" 01:05:46 DEBUG "value": [ 01:05:46 DEBUG "1915.0666666666666" 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG }, 01:05:46 DEBUG 1713485146, 01:05:46 DEBUG ] 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG "value": [ 01:05:46 DEBUG "0.000033333333333333335" 01:05:46 DEBUG } 01:05:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:05:46 DEBUG 01:05:46 DEBUG 01:05:46 DEBUG 1713485146, 01:05:46 DEBUG ] 01:05:46 DEBUG } 01:05:46 DEBUG ] 01:05:46 DEBUG [...] 01:05:46 DEBUG "instance": "10.142.0.43:9100" 01:05:46 DEBUG "1914.6333333333332" 01:05:46 DEBUG }, 01:05:46 DEBUG ] 01:05:46 DEBUG } 01:05:46 DEBUG 01:05:46 DEBUG }, 01:05:46 DEBUG ] 01:05:46 DEBUG "value": [ 01:05:46 DEBUG } 01:05:46 DEBUG } 01:05:46 DEBUG 01:05:46 DEBUG "value": [ 01:05:46 DEBUG } 01:05:46 DEBUG 1713485146, 01:05:46 DEBUG } 01:05:46 DEBUG 01:05:46 DEBUG 1713485146, 01:05:46 DEBUG ] 01:05:46 DEBUG "1070006272" 01:05:46 DEBUG "instance": "10.142.0.43:9100" 01:05:46 DEBUG "47261111" 01:05:46 DEBUG } 01:05:46 DEBUG ] 01:05:46 DEBUG }, 01:05:46 DEBUG ] 01:05:46 DEBUG } 01:05:46 DEBUG } 01:05:46 DEBUG "value": [ 01:05:46 DEBUG } 01:05:46 DEBUG ] 01:05:46 DEBUG 1713485146, 01:05:46 DEBUG ] 01:05:46 DEBUG } 01:05:46 DEBUG "1.0679999999999998" 01:05:46 DEBUG } 01:05:46 DEBUG } 01:05:46 DEBUG ] 01:05:46 DEBUG } 01:05:46 DEBUG } 01:05:46 DEBUG ] 01:05:46 DEBUG } 01:05:46 DEBUG } 01:05:46 INFO 01:05:46 INFO [loop_until]: kubectl --namespace=xlou top pods 01:05:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:05:46 INFO [loop_until]: OK (rc = 0) 01:05:46 DEBUG --- stdout --- 01:05:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10405m 9392Mi am-6fd554f8f7-hvvkw 8374m 8879Mi am-6fd554f8f7-pmj4b 8943m 9034Mi ds-cts-0 9036m 12298Mi ds-cts-1 9045m 12290Mi ds-cts-2 8265m 12373Mi ds-idrepo-0 4759m 3257Mi ds-idrepo-1 4793m 3122Mi ds-idrepo-2 4496m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 10m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6850m 926Mi 01:05:46 DEBUG --- stderr --- 01:05:46 DEBUG 01:05:46 INFO 01:05:46 INFO [loop_until]: kubectl --namespace=xlou top node 01:05:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:05:46 INFO [loop_until]: OK (rc = 0) 01:05:46 DEBUG --- stdout --- 01:05:46 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5913Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4493m 15% 6659Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8373m 28% 11616Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3223Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9093m 30% 14772Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 138m 0% 5224Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10515m 35% 12185Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27738m 92% 3134Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9600m 32% 11246Mi 10% gke-xlou-cdm-ds-0f601751-j488 4841m 16% 6081Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8860m 29% 16301Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5209m 17% 6746Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8315m 27% 14267Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7011m 44% 4666Mi 7% 01:05:46 DEBUG --- stderr --- 01:05:46 DEBUG 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485205" 01:06:45 INFO [http_cmd]: http status code OK 01:06:45 DEBUG --- status code --- 01:06:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:45 DEBUG --- http response --- 01:06:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:45 DEBUG ----- output ----- 01:06:45 DEBUG { 01:06:45 DEBUG "status": "success", 01:06:45 DEBUG "data": { 01:06:45 DEBUG "resultType": "vector", 01:06:45 DEBUG "result": [ 01:06:45 DEBUG { 01:06:45 DEBUG "metric": { 01:06:45 DEBUG "instance": "10.142.0.107:9100" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG [...] 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "instance": "10.142.0.43:9100" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 1713485205, 01:06:45 DEBUG "0.9681333333333346" 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485205" 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485205" 01:06:45 ERROR [http_cmd]: ERROR 01:06:45 ERROR --- status code --- 01:06:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:06:45 ERROR --- http response --- 01:06:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:06:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485205" 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485205" 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485205" 01:06:45 INFO [http_cmd]: http status code OK 01:06:45 DEBUG --- status code --- 01:06:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:45 DEBUG --- http response --- 01:06:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:06:45 INFO [http_cmd]: http status code OK 01:06:45 DEBUG --- status code --- 01:06:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:45 DEBUG --- http response --- 01:06:45 INFO [http_cmd]: http status code OK 01:06:45 DEBUG --- status code --- 01:06:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:45 DEBUG --- http response --- 01:06:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:45 INFO [http_cmd]: http status code OK 01:06:45 DEBUG ----- output ----- 01:06:45 DEBUG ----- output ----- 01:06:45 DEBUG --- status code --- 01:06:45 DEBUG { 01:06:45 DEBUG { 01:06:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:45 DEBUG "status": "success", 01:06:45 DEBUG "status": "success", 01:06:45 DEBUG --- http response --- 01:06:45 DEBUG "data": { 01:06:45 DEBUG "data": { 01:06:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:45 DEBUG "resultType": "vector", 01:06:45 DEBUG "resultType": "vector", 01:06:45 DEBUG ----- output ----- 01:06:45 DEBUG "result": [ 01:06:45 DEBUG "result": [ 01:06:45 DEBUG { 01:06:45 DEBUG { 01:06:45 DEBUG { 01:06:45 DEBUG "status": "success", 01:06:45 DEBUG "metric": { 01:06:45 DEBUG "metric": { 01:06:45 DEBUG "instance": "10.142.0.107:9100" 01:06:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:06:45 DEBUG "data": { 01:06:45 DEBUG }, 01:06:45 DEBUG }, 01:06:45 DEBUG "resultType": "vector", 01:06:45 DEBUG "value": [ 01:06:45 DEBUG "value": [ 01:06:45 DEBUG "result": [ 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG { 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "metric": { 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:06:45 DEBUG [...] 01:06:45 DEBUG [...] 01:06:45 DEBUG }, 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "instance": "10.142.0.43:9100" 01:06:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:06:45 DEBUG 01:06:45 DEBUG }, 01:06:45 DEBUG }, 01:06:45 DEBUG [...] 01:06:45 DEBUG "value": [ 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 01:06:45 DEBUG 1713485205, 01:06:45 DEBUG 1713485205, 01:06:45 DEBUG 01:06:45 DEBUG "326601044992" 01:06:45 DEBUG "0" 01:06:45 DEBUG 01:06:45 DEBUG ] 01:06:45 DEBUG ] 01:06:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:45 DEBUG }, 01:06:45 DEBUG ] 01:06:45 DEBUG ] 01:06:45 DEBUG "value": [ 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:45 DEBUG 1713485205, 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:45 DEBUG "433.0700535073793" 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485205" 01:06:45 INFO [http_cmd]: http status code OK 01:06:45 DEBUG --- status code --- 01:06:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:45 DEBUG --- http response --- 01:06:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:06:45 DEBUG ----- output ----- 01:06:45 DEBUG { 01:06:45 DEBUG "status": "success", 01:06:45 DEBUG "data": { 01:06:45 DEBUG "resultType": "vector", 01:06:45 DEBUG "result": [ 01:06:45 DEBUG { 01:06:45 DEBUG "metric": { 01:06:45 DEBUG "pod": "ds-idrepo-0" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG [...] 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "pod": "ds-cts-2" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 1713485205, 01:06:45 DEBUG "0" 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485205" 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485205" 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485205" 01:06:45 INFO [http_cmd]: http status code OK 01:06:45 INFO [http_cmd]: http status code OK 01:06:45 DEBUG --- status code --- 01:06:45 DEBUG --- status code --- 01:06:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:45 DEBUG --- http response --- 01:06:45 DEBUG --- http response --- 01:06:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:06:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:45 DEBUG ----- output ----- 01:06:45 DEBUG ----- output ----- 01:06:45 DEBUG { 01:06:45 DEBUG { 01:06:45 DEBUG "status": "success", 01:06:45 DEBUG "status": "success", 01:06:45 DEBUG "data": { 01:06:45 DEBUG "data": { 01:06:45 DEBUG "resultType": "vector", 01:06:45 DEBUG "resultType": "vector", 01:06:45 DEBUG "result": [ 01:06:45 DEBUG "result": [ 01:06:45 DEBUG { 01:06:45 DEBUG { 01:06:45 DEBUG "metric": { 01:06:45 INFO [http_cmd]: http status code OK 01:06:45 DEBUG "metric": { 01:06:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:06:45 DEBUG --- status code --- 01:06:45 DEBUG "instance": "10.142.0.107:9100" 01:06:45 DEBUG }, 01:06:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG --- http response --- 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG ----- output ----- 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG { 01:06:45 DEBUG [...] 01:06:45 DEBUG [...] 01:06:45 DEBUG "status": "success", 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "data": { 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "resultType": "vector", 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "result": [ 01:06:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:06:45 DEBUG "instance": "10.142.0.43:9100" 01:06:45 DEBUG { 01:06:45 DEBUG }, 01:06:45 DEBUG }, 01:06:45 DEBUG "metric": { 01:06:45 DEBUG "value": [ 01:06:45 DEBUG "value": [ 01:06:45 DEBUG "pod": "ds-cts-0" 01:06:45 DEBUG 1713485205, 01:06:45 DEBUG 1713485205, 01:06:45 DEBUG }, 01:06:45 DEBUG "4222.433021696948" 01:06:45 DEBUG "47255065.36666667" 01:06:45 DEBUG "value": [ 01:06:45 DEBUG ] 01:06:45 DEBUG ] 01:06:45 DEBUG 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:45 DEBUG 01:06:45 DEBUG ] 01:06:45 DEBUG ] 01:06:45 DEBUG 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:45 DEBUG [...] 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "pod": "ds-cts-2" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 1713485205, 01:06:45 DEBUG "0" 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485205" 01:06:45 INFO [http_cmd]: http status code OK 01:06:45 DEBUG --- status code --- 01:06:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:45 DEBUG --- http response --- 01:06:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:45 DEBUG ----- output ----- 01:06:45 DEBUG { 01:06:45 DEBUG "status": "success", 01:06:45 DEBUG "data": { 01:06:45 DEBUG "resultType": "vector", 01:06:45 DEBUG "result": [ 01:06:45 DEBUG { 01:06:45 DEBUG "metric": { 01:06:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG [...] 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 1713485205, 01:06:45 DEBUG "1907.1333333333334" 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485205" 01:06:45 INFO [http_cmd]: http status code OK 01:06:45 DEBUG --- status code --- 01:06:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:45 DEBUG --- http response --- 01:06:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:45 DEBUG ----- output ----- 01:06:45 DEBUG { 01:06:45 DEBUG "status": "success", 01:06:45 DEBUG "data": { 01:06:45 DEBUG "resultType": "vector", 01:06:45 DEBUG "result": [ 01:06:45 DEBUG { 01:06:45 DEBUG "metric": { 01:06:45 DEBUG "pod": "ds-idrepo-0" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG [...] 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "pod": "ds-idrepo-2" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 1713485205, 01:06:45 DEBUG "0" 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485205" 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485205" 01:06:45 INFO [http_cmd]: http status code OK 01:06:45 DEBUG --- status code --- 01:06:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:45 DEBUG --- http response --- 01:06:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:45 DEBUG ----- output ----- 01:06:45 DEBUG { 01:06:45 DEBUG "status": "success", 01:06:45 DEBUG "data": { 01:06:45 DEBUG "resultType": "vector", 01:06:45 DEBUG "result": [ 01:06:45 DEBUG { 01:06:45 DEBUG "metric": { 01:06:45 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG [...] 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 1713485205, 01:06:45 DEBUG "6.882787813364425" 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:45 INFO [http_cmd]: http status code OK 01:06:45 DEBUG --- status code --- 01:06:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:45 DEBUG --- http response --- 01:06:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:06:45 DEBUG ----- output ----- 01:06:45 DEBUG { 01:06:45 DEBUG "status": "success", 01:06:45 DEBUG "data": { 01:06:45 DEBUG "resultType": "vector", 01:06:45 DEBUG "result": [ 01:06:45 DEBUG { 01:06:45 DEBUG "metric": { 01:06:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG [...] 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 1713485205, 01:06:45 DEBUG "5.377412008029753" 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485205" 01:06:45 INFO [http_cmd]: http status code OK 01:06:45 DEBUG --- status code --- 01:06:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:45 DEBUG --- http response --- 01:06:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:45 DEBUG ----- output ----- 01:06:45 DEBUG { 01:06:45 DEBUG "status": "success", 01:06:45 DEBUG "data": { 01:06:45 DEBUG "resultType": "vector", 01:06:45 DEBUG "result": [ 01:06:45 DEBUG { 01:06:45 DEBUG "metric": { 01:06:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG [...] 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG 01:06:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:06:45 DEBUG }, 01:06:45 DEBUG "value": [ 01:06:45 DEBUG 1713485205, 01:06:45 DEBUG "0" 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG ] 01:06:45 DEBUG } 01:06:45 DEBUG } 01:06:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485206" 01:06:46 INFO [http_cmd]: http status code OK 01:06:46 DEBUG --- status code --- 01:06:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:46 DEBUG --- http response --- 01:06:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:46 DEBUG ----- output ----- 01:06:46 DEBUG { 01:06:46 DEBUG "status": "success", 01:06:46 DEBUG "data": { 01:06:46 DEBUG "resultType": "vector", 01:06:46 DEBUG "result": [ 01:06:46 DEBUG { 01:06:46 DEBUG "metric": { 01:06:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG [...] 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 1713485206, 01:06:46 DEBUG "0" 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG } 01:06:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485206" 01:06:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485206" 01:06:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485206" 01:06:46 INFO [http_cmd]: http status code OK 01:06:46 INFO [http_cmd]: http status code OK 01:06:46 DEBUG --- status code --- 01:06:46 DEBUG --- status code --- 01:06:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:46 DEBUG --- http response --- 01:06:46 DEBUG --- http response --- 01:06:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:46 DEBUG ----- output ----- 01:06:46 DEBUG { 01:06:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:46 DEBUG "status": "success", 01:06:46 DEBUG ----- output ----- 01:06:46 DEBUG "data": { 01:06:46 DEBUG { 01:06:46 DEBUG "resultType": "vector", 01:06:46 DEBUG "status": "success", 01:06:46 DEBUG "result": [ 01:06:46 DEBUG "data": { 01:06:46 DEBUG { 01:06:46 DEBUG "resultType": "vector", 01:06:46 DEBUG "metric": { 01:06:46 DEBUG "result": [ 01:06:46 DEBUG "pod": "ds-idrepo-0" 01:06:46 DEBUG { 01:06:46 DEBUG }, 01:06:46 DEBUG "metric": { 01:06:46 DEBUG "value": [ 01:06:46 DEBUG "pod": "ds-idrepo-0" 01:06:46 DEBUG 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG [...] 01:06:46 DEBUG [...] 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG "pod": "ds-idrepo-2" 01:06:46 DEBUG "pod": "ds-idrepo-2" 01:06:46 DEBUG }, 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 1713485206, 01:06:46 DEBUG 1713485206, 01:06:46 DEBUG "0" 01:06:46 DEBUG "0" 01:06:46 DEBUG ] 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG } 01:06:46 DEBUG ] 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG } 01:06:46 DEBUG } 01:06:46 DEBUG } 01:06:46 INFO [http_cmd]: http status code OK 01:06:46 DEBUG --- status code --- 01:06:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:46 DEBUG --- http response --- 01:06:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:46 DEBUG ----- output ----- 01:06:46 DEBUG { 01:06:46 DEBUG "status": "success", 01:06:46 DEBUG "data": { 01:06:46 DEBUG "resultType": "vector", 01:06:46 DEBUG "result": [ 01:06:46 DEBUG { 01:06:46 DEBUG "metric": { 01:06:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG [...] 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 1713485206, 01:06:46 DEBUG "1907.6" 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG } 01:06:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485206" 01:06:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485206" 01:06:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485206" 01:06:46 INFO [http_cmd]: http status code OK 01:06:46 DEBUG --- status code --- 01:06:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:46 DEBUG --- http response --- 01:06:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:46 DEBUG ----- output ----- 01:06:46 DEBUG { 01:06:46 DEBUG "status": "success", 01:06:46 DEBUG "data": { 01:06:46 DEBUG "resultType": "vector", 01:06:46 DEBUG "result": [ 01:06:46 DEBUG { 01:06:46 DEBUG "metric": { 01:06:46 DEBUG "instance": "10.142.0.107:9100" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG [...] 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG "instance": "10.142.0.43:9100" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 1713485206, 01:06:46 DEBUG "47260038" 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG } 01:06:46 INFO [http_cmd]: http status code OK 01:06:46 DEBUG --- status code --- 01:06:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:46 DEBUG --- http response --- 01:06:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:46 DEBUG ----- output ----- 01:06:46 DEBUG { 01:06:46 DEBUG "status": "success", 01:06:46 DEBUG "data": { 01:06:46 DEBUG "resultType": "vector", 01:06:46 DEBUG "result": [ 01:06:46 DEBUG { 01:06:46 DEBUG "metric": { 01:06:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG [...] 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 1713485206, 01:06:46 DEBUG "2644427" 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG } 01:06:46 INFO [http_cmd]: http status code OK 01:06:46 DEBUG --- status code --- 01:06:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:46 DEBUG --- http response --- 01:06:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:46 DEBUG ----- output ----- 01:06:46 DEBUG { 01:06:46 DEBUG "status": "success", 01:06:46 DEBUG "data": { 01:06:46 DEBUG "resultType": "vector", 01:06:46 DEBUG "result": [ 01:06:46 DEBUG { 01:06:46 DEBUG "metric": { 01:06:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG [...] 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 1713485206, 01:06:46 DEBUG "1907.2666666666667" 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG } 01:06:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485206" 01:06:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485206" 01:06:46 INFO [http_cmd]: http status code OK 01:06:46 DEBUG --- status code --- 01:06:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:46 DEBUG --- http response --- 01:06:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:06:46 DEBUG ----- output ----- 01:06:46 DEBUG { 01:06:46 DEBUG "status": "success", 01:06:46 DEBUG "data": { 01:06:46 DEBUG "resultType": "vector", 01:06:46 DEBUG "result": [ 01:06:46 DEBUG { 01:06:46 DEBUG "metric": { 01:06:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG [...] 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485206" 01:06:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485206" 01:06:46 DEBUG 01:06:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485206" 01:06:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 1713485206, 01:06:46 DEBUG "20.076" 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG } 01:06:46 INFO [http_cmd]: http status code OK 01:06:46 DEBUG --- status code --- 01:06:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:46 DEBUG --- http response --- 01:06:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:06:46 DEBUG ----- output ----- 01:06:46 DEBUG { 01:06:46 DEBUG "status": "success", 01:06:46 DEBUG "data": { 01:06:46 DEBUG "resultType": "vector", 01:06:46 DEBUG "result": [ 01:06:46 DEBUG { 01:06:46 DEBUG "metric": { 01:06:46 DEBUG "pod": "ds-cts-0" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG [...] 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG "pod": "ds-cts-2" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 1713485206, 01:06:46 DEBUG "0.000033333333333333335" 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG } 01:06:46 INFO [http_cmd]: http status code OK 01:06:46 INFO [http_cmd]: http status code OK 01:06:46 DEBUG --- status code --- 01:06:46 DEBUG --- status code --- 01:06:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:46 DEBUG --- http response --- 01:06:46 DEBUG --- http response --- 01:06:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:06:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:46 DEBUG ----- output ----- 01:06:46 DEBUG ----- output ----- 01:06:46 DEBUG { 01:06:46 DEBUG { 01:06:46 DEBUG "status": "success", 01:06:46 DEBUG "status": "success", 01:06:46 DEBUG "data": { 01:06:46 DEBUG "data": { 01:06:46 DEBUG "resultType": "vector", 01:06:46 DEBUG "resultType": "vector", 01:06:46 DEBUG "result": [ 01:06:46 DEBUG "result": [ 01:06:46 DEBUG { 01:06:46 DEBUG { 01:06:46 DEBUG "metric": { 01:06:46 DEBUG "metric": { 01:06:46 DEBUG "pod": "ds-cts-0" 01:06:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:06:46 DEBUG }, 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG [...] 01:06:46 DEBUG [...] 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG "pod": "ds-cts-2" 01:06:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:06:46 DEBUG }, 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 1713485206, 01:06:46 DEBUG 1713485206, 01:06:46 DEBUG "1506.6666666666667" 01:06:46 DEBUG "1071800320" 01:06:46 DEBUG ] 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 INFO [http_cmd]: http status code OK 01:06:46 DEBUG } 01:06:46 DEBUG ] 01:06:46 DEBUG --- status code --- 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG } 01:06:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:06:46 DEBUG --- http response --- 01:06:46 DEBUG } 01:06:46 DEBUG } 01:06:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:06:46 DEBUG ----- output ----- 01:06:46 DEBUG { 01:06:46 DEBUG "status": "success", 01:06:46 DEBUG "data": { 01:06:46 DEBUG "resultType": "vector", 01:06:46 DEBUG "result": [ 01:06:46 DEBUG { 01:06:46 DEBUG "metric": { 01:06:46 DEBUG "instance": "10.142.0.107:9100" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG [...] 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG 01:06:46 DEBUG "instance": "10.142.0.43:9100" 01:06:46 DEBUG }, 01:06:46 DEBUG "value": [ 01:06:46 DEBUG 1713485206, 01:06:46 DEBUG "1.1469999999999994" 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG ] 01:06:46 DEBUG } 01:06:46 DEBUG } 01:06:46 INFO 01:06:46 INFO [loop_until]: kubectl --namespace=xlou top pods 01:06:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:06:46 INFO [loop_until]: OK (rc = 0) 01:06:46 DEBUG --- stdout --- 01:06:46 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10472m 9394Mi am-6fd554f8f7-hvvkw 8348m 8879Mi am-6fd554f8f7-pmj4b 8918m 9035Mi ds-cts-0 9155m 12376Mi ds-cts-1 8204m 12339Mi ds-cts-2 8651m 12344Mi ds-idrepo-0 4676m 3258Mi ds-idrepo-1 4588m 3125Mi ds-idrepo-2 4478m 3038Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 13m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6795m 928Mi 01:06:46 DEBUG --- stderr --- 01:06:46 DEBUG 01:06:47 INFO 01:06:47 INFO [loop_until]: kubectl --namespace=xlou top node 01:06:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:06:47 INFO [loop_until]: OK (rc = 0) 01:06:47 DEBUG --- stdout --- 01:06:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5930Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4604m 15% 6661Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8456m 28% 11606Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3229Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9267m 31% 14847Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5236Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10506m 35% 12190Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 26669m 89% 3136Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 9874m 33% 11261Mi 10% gke-xlou-cdm-ds-0f601751-j488 4949m 16% 6085Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8451m 28% 16341Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5214m 17% 6744Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8476m 28% 14258Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 6995m 44% 4666Mi 7% 01:06:47 DEBUG --- stderr --- 01:06:47 DEBUG 01:07:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485265" 01:07:45 INFO [http_cmd]: http status code OK 01:07:45 DEBUG --- status code --- 01:07:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:45 DEBUG --- http response --- 01:07:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:45 DEBUG ----- output ----- 01:07:45 DEBUG { 01:07:45 DEBUG "status": "success", 01:07:45 DEBUG "data": { 01:07:45 DEBUG "resultType": "vector", 01:07:45 DEBUG "result": [ 01:07:45 DEBUG { 01:07:45 DEBUG "metric": { 01:07:45 DEBUG "instance": "10.142.0.107:9100" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG [...] 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG "instance": "10.142.0.43:9100" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 1713485265, 01:07:45 DEBUG "0.9167999999999997" 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG } 01:07:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485265" 01:07:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485265" 01:07:45 ERROR [http_cmd]: ERROR 01:07:45 ERROR --- status code --- 01:07:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:07:45 ERROR --- http response --- 01:07:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:07:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:07:45 INFO [http_cmd]: http status code OK 01:07:45 DEBUG --- status code --- 01:07:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:45 DEBUG --- http response --- 01:07:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:07:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485265" 01:07:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485265" 01:07:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485265" 01:07:45 INFO [http_cmd]: http status code OK 01:07:45 DEBUG --- status code --- 01:07:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:45 DEBUG --- http response --- 01:07:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:45 DEBUG ----- output ----- 01:07:45 DEBUG { 01:07:45 DEBUG "status": "success", 01:07:45 DEBUG "data": { 01:07:45 DEBUG "resultType": "vector", 01:07:45 DEBUG "result": [ 01:07:45 DEBUG { 01:07:45 DEBUG "metric": { 01:07:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG [...] 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 1713485265, 01:07:45 DEBUG "0" 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG } 01:07:45 INFO [http_cmd]: http status code OK 01:07:45 DEBUG --- status code --- 01:07:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:45 DEBUG --- http response --- 01:07:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:07:45 DEBUG ----- output ----- 01:07:45 DEBUG { 01:07:45 DEBUG "status": "success", 01:07:45 DEBUG "data": { 01:07:45 DEBUG "resultType": "vector", 01:07:45 DEBUG "result": [ 01:07:45 DEBUG { 01:07:45 DEBUG "metric": { 01:07:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG [...] 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 1713485265, 01:07:45 DEBUG "0" 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG ] 01:07:45 INFO [http_cmd]: http status code OK 01:07:45 DEBUG } 01:07:45 DEBUG --- status code --- 01:07:45 DEBUG } 01:07:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:45 DEBUG --- http response --- 01:07:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:45 DEBUG ----- output ----- 01:07:45 DEBUG { 01:07:45 DEBUG "status": "success", 01:07:45 DEBUG "data": { 01:07:45 DEBUG "resultType": "vector", 01:07:45 DEBUG "result": [ 01:07:45 DEBUG { 01:07:45 DEBUG "metric": { 01:07:45 DEBUG "instance": "10.142.0.107:9100" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG [...] 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG "instance": "10.142.0.43:9100" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 1713485265, 01:07:45 DEBUG "328985922560" 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG } 01:07:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485265" 01:07:45 INFO [http_cmd]: http status code OK 01:07:45 DEBUG --- status code --- 01:07:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:45 DEBUG --- http response --- 01:07:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:07:45 DEBUG ----- output ----- 01:07:45 DEBUG { 01:07:45 DEBUG "status": "success", 01:07:45 DEBUG "data": { 01:07:45 DEBUG "resultType": "vector", 01:07:45 DEBUG "result": [ 01:07:45 DEBUG { 01:07:45 DEBUG "metric": { 01:07:45 DEBUG "pod": "ds-idrepo-0" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG [...] 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG "pod": "ds-cts-2" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 1713485265, 01:07:45 DEBUG "0" 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG } 01:07:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485265" 01:07:45 INFO [http_cmd]: http status code OK 01:07:45 DEBUG --- status code --- 01:07:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:45 DEBUG --- http response --- 01:07:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:45 DEBUG ----- output ----- 01:07:45 DEBUG { 01:07:45 DEBUG "status": "success", 01:07:45 DEBUG "data": { 01:07:45 DEBUG "resultType": "vector", 01:07:45 DEBUG "result": [ 01:07:45 DEBUG { 01:07:45 DEBUG "metric": { 01:07:45 DEBUG "instance": "10.142.0.107:9100" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG [...] 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG "instance": "10.142.0.43:9100" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 1713485265, 01:07:45 DEBUG "47401448.33333333" 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG } 01:07:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485265" 01:07:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485265" 01:07:45 INFO [http_cmd]: http status code OK 01:07:45 INFO [http_cmd]: http status code OK 01:07:45 DEBUG --- status code --- 01:07:45 DEBUG --- status code --- 01:07:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:45 DEBUG --- http response --- 01:07:45 DEBUG --- http response --- 01:07:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:45 DEBUG ----- output ----- 01:07:45 DEBUG ----- output ----- 01:07:45 DEBUG { 01:07:45 DEBUG { 01:07:45 DEBUG "status": "success", 01:07:45 DEBUG "status": "success", 01:07:45 DEBUG "data": { 01:07:45 DEBUG "data": { 01:07:45 DEBUG "resultType": "vector", 01:07:45 DEBUG "resultType": "vector", 01:07:45 DEBUG "result": [ 01:07:45 DEBUG "result": [ 01:07:45 DEBUG { 01:07:45 DEBUG { 01:07:45 DEBUG "metric": { 01:07:45 DEBUG "metric": { 01:07:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:07:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:07:45 DEBUG }, 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485265" 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG [...] 01:07:45 DEBUG [...] 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG "pod": "ds-idrepo-0" 01:07:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:07:45 DEBUG }, 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 1713485265, 01:07:45 DEBUG 1713485265, 01:07:45 DEBUG "0" 01:07:45 DEBUG "1911.1666666666667" 01:07:45 DEBUG ] 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG } 01:07:45 DEBUG ] 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG } 01:07:45 DEBUG } 01:07:45 DEBUG } 01:07:45 INFO [http_cmd]: http status code OK 01:07:45 DEBUG --- status code --- 01:07:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:45 DEBUG --- http response --- 01:07:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:07:45 DEBUG ----- output ----- 01:07:45 DEBUG { 01:07:45 DEBUG "status": "success", 01:07:45 DEBUG "data": { 01:07:45 DEBUG "resultType": "vector", 01:07:45 DEBUG "result": [ 01:07:45 DEBUG { 01:07:45 DEBUG "metric": { 01:07:45 DEBUG "pod": "ds-cts-0" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG [...] 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG "pod": "ds-cts-2" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 1713485265, 01:07:45 DEBUG "0" 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG } 01:07:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485265" 01:07:45 INFO [http_cmd]: http status code OK 01:07:45 DEBUG --- status code --- 01:07:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:45 DEBUG --- http response --- 01:07:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:45 DEBUG ----- output ----- 01:07:45 DEBUG { 01:07:45 DEBUG "status": "success", 01:07:45 DEBUG "data": { 01:07:45 DEBUG "resultType": "vector", 01:07:45 DEBUG "result": [ 01:07:45 DEBUG { 01:07:45 DEBUG "metric": { 01:07:45 DEBUG "pod": "ds-idrepo-0" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG [...] 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG 01:07:45 DEBUG "pod": "ds-idrepo-2" 01:07:45 DEBUG }, 01:07:45 DEBUG "value": [ 01:07:45 DEBUG 1713485265, 01:07:45 DEBUG "0" 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG ] 01:07:45 DEBUG } 01:07:45 DEBUG } 01:07:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485265" 01:07:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485265" 01:07:45 INFO [http_cmd]: http status code OK 01:07:45 DEBUG --- status code --- 01:07:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:45 DEBUG --- http response --- 01:07:46 INFO [http_cmd]: http status code OK 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:46 DEBUG --- status code --- 01:07:46 DEBUG ----- output ----- 01:07:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:46 DEBUG { 01:07:46 DEBUG --- http response --- 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:46 DEBUG "data": { 01:07:46 DEBUG ----- output ----- 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG { 01:07:46 DEBUG "result": [ 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG { 01:07:46 DEBUG "data": { 01:07:46 DEBUG "metric": { 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:07:46 DEBUG "result": [ 01:07:46 DEBUG }, 01:07:46 DEBUG { 01:07:46 DEBUG "value": [ 01:07:46 DEBUG "metric": { 01:07:46 DEBUG 01:07:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:07:46 DEBUG 01:07:46 DEBUG }, 01:07:46 DEBUG 01:07:46 DEBUG "value": [ 01:07:46 DEBUG [...] 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG [...] 01:07:46 DEBUG "pod": "ds-idrepo-0" 01:07:46 DEBUG 01:07:46 DEBUG }, 01:07:46 DEBUG 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 01:07:46 DEBUG 1713485265, 01:07:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:07:46 DEBUG "0" 01:07:46 DEBUG }, 01:07:46 DEBUG ] 01:07:46 DEBUG "value": [ 01:07:46 DEBUG } 01:07:46 DEBUG 1713485265, 01:07:46 DEBUG ] 01:07:46 DEBUG "6.886377429993656" 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG } 01:07:46 DEBUG } 01:07:46 DEBUG } 01:07:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485266" 01:07:46 INFO [http_cmd]: http status code OK 01:07:46 DEBUG --- status code --- 01:07:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:46 DEBUG --- http response --- 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:46 DEBUG ----- output ----- 01:07:46 DEBUG { 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG "data": { 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG "result": [ 01:07:46 DEBUG { 01:07:46 DEBUG "metric": { 01:07:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG [...] 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 1713485266, 01:07:46 DEBUG "0" 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG } 01:07:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485266" 01:07:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485266" 01:07:46 INFO [http_cmd]: http status code OK 01:07:46 INFO [http_cmd]: http status code OK 01:07:46 DEBUG --- status code --- 01:07:46 DEBUG --- status code --- 01:07:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:46 DEBUG --- http response --- 01:07:46 DEBUG --- http response --- 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:46 DEBUG ----- output ----- 01:07:46 DEBUG { 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG "data": { 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG "result": [ 01:07:46 DEBUG { 01:07:46 DEBUG "metric": { 01:07:46 DEBUG "pod": "ds-idrepo-0" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG [...] 01:07:46 DEBUG 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:07:46 DEBUG 01:07:46 DEBUG ----- output ----- 01:07:46 DEBUG 01:07:46 DEBUG { 01:07:46 DEBUG "pod": "ds-idrepo-2" 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG }, 01:07:46 DEBUG "data": { 01:07:46 DEBUG "value": [ 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG 1713485266, 01:07:46 DEBUG "result": [ 01:07:46 DEBUG "0" 01:07:46 DEBUG { 01:07:46 DEBUG ] 01:07:46 DEBUG "metric": { 01:07:46 DEBUG } 01:07:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:07:46 DEBUG ] 01:07:46 DEBUG }, 01:07:46 DEBUG } 01:07:46 DEBUG "value": [ 01:07:46 DEBUG } 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG [...] 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 1713485266, 01:07:46 DEBUG "0" 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG } 01:07:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485266" 01:07:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485266" 01:07:46 INFO [http_cmd]: http status code OK 01:07:46 DEBUG --- status code --- 01:07:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:46 DEBUG --- http response --- 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:46 DEBUG ----- output ----- 01:07:46 DEBUG { 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG "data": { 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG "result": [ 01:07:46 DEBUG { 01:07:46 DEBUG "metric": { 01:07:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG [...] 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 1713485266, 01:07:46 DEBUG "1911.2666666666667" 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG } 01:07:46 INFO [http_cmd]: http status code OK 01:07:46 DEBUG --- status code --- 01:07:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:46 DEBUG --- http response --- 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:46 DEBUG ----- output ----- 01:07:46 DEBUG { 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG "data": { 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG "result": [ 01:07:46 DEBUG { 01:07:46 DEBUG "metric": { 01:07:46 DEBUG "pod": "ds-idrepo-0" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG [...] 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG "pod": "ds-idrepo-2" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 1713485266, 01:07:46 DEBUG "0" 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG } 01:07:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485266" 01:07:46 INFO [http_cmd]: http status code OK 01:07:46 DEBUG --- status code --- 01:07:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:46 DEBUG --- http response --- 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:46 DEBUG ----- output ----- 01:07:46 DEBUG { 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG "data": { 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG "result": [ 01:07:46 DEBUG { 01:07:46 DEBUG "metric": { 01:07:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG [...] 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 1713485266, 01:07:46 DEBUG "2759194" 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG } 01:07:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485266" 01:07:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485266" 01:07:46 INFO [http_cmd]: http status code OK 01:07:46 DEBUG --- status code --- 01:07:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:46 DEBUG --- http response --- 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:46 DEBUG ----- output ----- 01:07:46 DEBUG { 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG "data": { 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG "result": [ 01:07:46 DEBUG { 01:07:46 DEBUG "metric": { 01:07:46 DEBUG "instance": "10.142.0.107:9100" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG [...] 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG "instance": "10.142.0.43:9100" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 1713485266, 01:07:46 DEBUG "47405459.33333333" 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG } 01:07:46 INFO [http_cmd]: http status code OK 01:07:46 DEBUG --- status code --- 01:07:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:46 DEBUG --- http response --- 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:46 DEBUG ----- output ----- 01:07:46 DEBUG { 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG "data": { 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG "result": [ 01:07:46 DEBUG { 01:07:46 DEBUG "metric": { 01:07:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG [...] 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 1713485266, 01:07:46 DEBUG "1911" 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG } 01:07:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485266" 01:07:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485266" 01:07:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485266" 01:07:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485266" 01:07:46 INFO [http_cmd]: http status code OK 01:07:46 INFO [http_cmd]: http status code OK 01:07:46 DEBUG --- status code --- 01:07:46 DEBUG --- status code --- 01:07:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:46 DEBUG --- http response --- 01:07:46 DEBUG --- http response --- 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:07:46 DEBUG ----- output ----- 01:07:46 DEBUG ----- output ----- 01:07:46 DEBUG { 01:07:46 INFO [http_cmd]: http status code OK 01:07:46 DEBUG { 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG --- status code --- 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG "data": { 01:07:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:46 DEBUG "data": { 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG --- http response --- 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG "result": [ 01:07:46 DEBUG "result": [ 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:46 DEBUG { 01:07:46 INFO [http_cmd]: http status code OK 01:07:46 DEBUG { 01:07:46 DEBUG ----- output ----- 01:07:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485266" 01:07:46 DEBUG "metric": { 01:07:46 DEBUG --- status code --- 01:07:46 DEBUG "metric": { 01:07:46 DEBUG { 01:07:46 DEBUG "pod": "ds-cts-0" 01:07:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG }, 01:07:46 DEBUG --- http response --- 01:07:46 DEBUG }, 01:07:46 DEBUG "data": { 01:07:46 DEBUG "value": [ 01:07:46 DEBUG "value": [ 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG ----- output ----- 01:07:46 DEBUG "result": [ 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG { 01:07:46 DEBUG { 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG "metric": { 01:07:46 DEBUG [...] 01:07:46 DEBUG [...] 01:07:46 DEBUG "data": { 01:07:46 DEBUG "instance": "10.142.0.107:9100" 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG }, 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG "result": [ 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG { 01:07:46 DEBUG 01:07:46 DEBUG "pod": "ds-cts-2" 01:07:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:07:46 DEBUG "metric": { 01:07:46 DEBUG 01:07:46 DEBUG }, 01:07:46 DEBUG }, 01:07:46 DEBUG "pod": "ds-cts-0" 01:07:46 DEBUG 01:07:46 DEBUG "value": [ 01:07:46 DEBUG "value": [ 01:07:46 DEBUG }, 01:07:46 DEBUG [...] 01:07:46 DEBUG 1713485266, 01:07:46 DEBUG 1713485266, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 01:07:46 DEBUG "3451.8333333333335" 01:07:46 DEBUG "20.076" 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG ] 01:07:46 DEBUG ] 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG "instance": "10.142.0.43:9100" 01:07:46 DEBUG } 01:07:46 DEBUG 01:07:46 DEBUG } 01:07:46 DEBUG }, 01:07:46 DEBUG ] 01:07:46 DEBUG [...] 01:07:46 DEBUG ] 01:07:46 DEBUG "value": [ 01:07:46 DEBUG } 01:07:46 DEBUG 01:07:46 DEBUG } 01:07:46 DEBUG 1713485266, 01:07:46 DEBUG } 01:07:46 DEBUG 01:07:46 DEBUG } 01:07:46 DEBUG "0.8526666666666672" 01:07:46 DEBUG 01:07:46 DEBUG ] 01:07:46 DEBUG "pod": "ds-cts-2" 01:07:46 DEBUG } 01:07:46 DEBUG }, 01:07:46 DEBUG ] 01:07:46 DEBUG "value": [ 01:07:46 DEBUG } 01:07:46 DEBUG 1713485266, 01:07:46 DEBUG } 01:07:46 DEBUG "0.00006666666666666667" 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG } 01:07:46 INFO [http_cmd]: http status code OK 01:07:46 DEBUG --- status code --- 01:07:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:07:46 DEBUG --- http response --- 01:07:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:07:46 DEBUG ----- output ----- 01:07:46 DEBUG { 01:07:46 DEBUG "status": "success", 01:07:46 DEBUG "data": { 01:07:46 DEBUG "resultType": "vector", 01:07:46 DEBUG "result": [ 01:07:46 DEBUG { 01:07:46 DEBUG "metric": { 01:07:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG [...] 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG 01:07:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:07:46 DEBUG }, 01:07:46 DEBUG "value": [ 01:07:46 DEBUG 1713485266, 01:07:46 DEBUG "1073836032" 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG ] 01:07:46 DEBUG } 01:07:46 DEBUG } 01:07:46 INFO 01:07:46 INFO [loop_until]: kubectl --namespace=xlou top pods 01:07:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:07:47 INFO [loop_until]: OK (rc = 0) 01:07:47 DEBUG --- stdout --- 01:07:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10927m 9396Mi am-6fd554f8f7-hvvkw 8635m 8879Mi am-6fd554f8f7-pmj4b 9174m 9036Mi ds-cts-0 8914m 12401Mi ds-cts-1 8696m 12377Mi ds-cts-2 9610m 12466Mi ds-idrepo-0 4669m 3257Mi ds-idrepo-1 4745m 3125Mi ds-idrepo-2 4592m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6969m 930Mi 01:07:47 DEBUG --- stderr --- 01:07:47 DEBUG 01:07:47 INFO 01:07:47 INFO [loop_until]: kubectl --namespace=xlou top node 01:07:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:07:47 INFO [loop_until]: OK (rc = 0) 01:07:47 DEBUG --- stdout --- 01:07:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5928Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4650m 15% 6656Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8480m 28% 11612Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3229Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9063m 30% 14846Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 5232Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10765m 36% 12182Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28000m 93% 3135Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10023m 33% 11253Mi 10% gke-xlou-cdm-ds-0f601751-j488 4984m 16% 6091Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8762m 29% 16376Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5249m 17% 6760Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9629m 32% 14377Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7120m 44% 4668Mi 7% 01:07:47 DEBUG --- stderr --- 01:07:47 DEBUG 01:08:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485325" 01:08:45 INFO [http_cmd]: http status code OK 01:08:45 DEBUG --- status code --- 01:08:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:45 DEBUG --- http response --- 01:08:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:08:45 DEBUG ----- output ----- 01:08:45 DEBUG { 01:08:45 DEBUG "status": "success", 01:08:45 DEBUG "data": { 01:08:45 DEBUG "resultType": "vector", 01:08:45 DEBUG "result": [ 01:08:45 DEBUG { 01:08:45 DEBUG "metric": { 01:08:45 DEBUG "instance": "10.142.0.107:9100" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG [...] 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG "instance": "10.142.0.43:9100" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 1713485325, 01:08:45 DEBUG "0.9736666666666679" 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG } 01:08:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485325" 01:08:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485325" 01:08:45 ERROR [http_cmd]: ERROR 01:08:45 ERROR --- status code --- 01:08:45 INFO [http_cmd]: http status code OK 01:08:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:08:45 DEBUG --- status code --- 01:08:45 ERROR --- http response --- 01:08:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:08:45 DEBUG --- http response --- 01:08:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:08:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:08:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485325" 01:08:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485325" 01:08:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485325" 01:08:45 INFO [http_cmd]: http status code OK 01:08:45 DEBUG --- status code --- 01:08:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:45 DEBUG --- http response --- 01:08:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:45 DEBUG ----- output ----- 01:08:45 DEBUG { 01:08:45 DEBUG "status": "success", 01:08:45 DEBUG "data": { 01:08:45 DEBUG "resultType": "vector", 01:08:45 DEBUG "result": [ 01:08:45 DEBUG { 01:08:45 DEBUG "metric": { 01:08:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG [...] 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 1713485325, 01:08:45 DEBUG "0" 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG } 01:08:45 INFO [http_cmd]: http status code OK 01:08:45 DEBUG --- status code --- 01:08:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:45 DEBUG --- http response --- 01:08:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:08:45 DEBUG ----- output ----- 01:08:45 DEBUG { 01:08:45 DEBUG "status": "success", 01:08:45 DEBUG "data": { 01:08:45 DEBUG "resultType": "vector", 01:08:45 DEBUG "result": [ 01:08:45 DEBUG { 01:08:45 DEBUG "metric": { 01:08:45 DEBUG "instance": "10.142.0.107:9100" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG [...] 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG "instance": "10.142.0.43:9100" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 1713485325, 01:08:45 DEBUG "331409132544" 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG } 01:08:45 INFO [http_cmd]: http status code OK 01:08:45 DEBUG --- status code --- 01:08:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:45 DEBUG --- http response --- 01:08:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:08:45 DEBUG ----- output ----- 01:08:45 DEBUG { 01:08:45 DEBUG "status": "success", 01:08:45 DEBUG "data": { 01:08:45 DEBUG "resultType": "vector", 01:08:45 DEBUG "result": [ 01:08:45 DEBUG { 01:08:45 DEBUG "metric": { 01:08:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG [...] 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 1713485325, 01:08:45 DEBUG "0" 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG } 01:08:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485325" 01:08:45 INFO [http_cmd]: http status code OK 01:08:45 DEBUG --- status code --- 01:08:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:45 DEBUG --- http response --- 01:08:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:08:45 DEBUG ----- output ----- 01:08:45 DEBUG { 01:08:45 DEBUG "status": "success", 01:08:45 DEBUG "data": { 01:08:45 DEBUG "resultType": "vector", 01:08:45 DEBUG "result": [ 01:08:45 DEBUG { 01:08:45 DEBUG "metric": { 01:08:45 DEBUG "pod": "ds-idrepo-0" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG [...] 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG "pod": "ds-cts-2" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 1713485325, 01:08:45 DEBUG "0" 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG } 01:08:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485325" 01:08:45 INFO [http_cmd]: http status code OK 01:08:45 DEBUG --- status code --- 01:08:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:45 DEBUG --- http response --- 01:08:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:08:45 DEBUG ----- output ----- 01:08:45 DEBUG { 01:08:45 DEBUG "status": "success", 01:08:45 DEBUG "data": { 01:08:45 DEBUG "resultType": "vector", 01:08:45 DEBUG "result": [ 01:08:45 DEBUG { 01:08:45 DEBUG "metric": { 01:08:45 DEBUG "instance": "10.142.0.107:9100" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG [...] 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG "instance": "10.142.0.43:9100" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 1713485325, 01:08:45 DEBUG "47289600.96666667" 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG } 01:08:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485325" 01:08:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485325" 01:08:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485325" 01:08:45 INFO [http_cmd]: http status code OK 01:08:45 DEBUG --- status code --- 01:08:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:45 DEBUG --- http response --- 01:08:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:08:45 DEBUG ----- output ----- 01:08:45 DEBUG { 01:08:45 DEBUG "status": "success", 01:08:45 DEBUG "data": { 01:08:45 DEBUG "resultType": "vector", 01:08:45 DEBUG "result": [ 01:08:45 DEBUG { 01:08:45 DEBUG "metric": { 01:08:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG [...] 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 1713485325, 01:08:45 DEBUG "35894771.80697387" 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG } 01:08:45 INFO [http_cmd]: http status code OK 01:08:45 DEBUG --- status code --- 01:08:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:45 DEBUG --- http response --- 01:08:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:45 DEBUG ----- output ----- 01:08:45 DEBUG { 01:08:45 DEBUG "status": "success", 01:08:45 DEBUG "data": { 01:08:45 DEBUG "resultType": "vector", 01:08:45 DEBUG "result": [ 01:08:45 DEBUG { 01:08:45 DEBUG "metric": { 01:08:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG [...] 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 1713485325, 01:08:45 DEBUG "1937.1333333333334" 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG } 01:08:45 INFO [http_cmd]: http status code OK 01:08:45 DEBUG --- status code --- 01:08:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:45 DEBUG --- http response --- 01:08:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:08:45 DEBUG ----- output ----- 01:08:45 DEBUG { 01:08:45 DEBUG "status": "success", 01:08:45 DEBUG "data": { 01:08:45 DEBUG "resultType": "vector", 01:08:45 DEBUG "result": [ 01:08:45 DEBUG { 01:08:45 DEBUG "metric": { 01:08:45 DEBUG "pod": "ds-cts-0" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG [...] 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG "pod": "ds-cts-2" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 1713485325, 01:08:45 DEBUG "0" 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG } 01:08:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485325" 01:08:45 INFO [http_cmd]: http status code OK 01:08:45 DEBUG --- status code --- 01:08:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:45 DEBUG --- http response --- 01:08:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:45 DEBUG ----- output ----- 01:08:45 DEBUG { 01:08:45 DEBUG "status": "success", 01:08:45 DEBUG "data": { 01:08:45 DEBUG "resultType": "vector", 01:08:45 DEBUG "result": [ 01:08:45 DEBUG { 01:08:45 DEBUG "metric": { 01:08:45 DEBUG "pod": "ds-idrepo-0" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG [...] 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG 01:08:45 DEBUG "pod": "ds-idrepo-2" 01:08:45 DEBUG }, 01:08:45 DEBUG "value": [ 01:08:45 DEBUG 1713485325, 01:08:45 DEBUG "0" 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG ] 01:08:45 DEBUG } 01:08:45 DEBUG } 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485326" 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485326" 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG --- http response --- 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG { 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG "data": { 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG "result": [ 01:08:46 DEBUG { 01:08:46 DEBUG "metric": { 01:08:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG [...] 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG "6.979153758960002" 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG ] 01:08:46 DEBUG --- http response --- 01:08:46 DEBUG } 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:08:46 DEBUG ] 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG } 01:08:46 DEBUG { 01:08:46 DEBUG } 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG "data": { 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG "result": [ 01:08:46 DEBUG { 01:08:46 DEBUG "metric": { 01:08:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG [...] 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG "13.857559056054445" 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG } 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485326" 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG --- http response --- 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG { 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG "data": { 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG "result": [ 01:08:46 DEBUG { 01:08:46 DEBUG "metric": { 01:08:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG [...] 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG "0" 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG } 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485326" 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485326" 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG --- http response --- 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG { 01:08:46 DEBUG --- http response --- 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:08:46 DEBUG "data": { 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG { 01:08:46 DEBUG "result": [ 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG { 01:08:46 DEBUG "data": { 01:08:46 DEBUG "metric": { 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG "pod": "ds-idrepo-0" 01:08:46 DEBUG "result": [ 01:08:46 DEBUG }, 01:08:46 DEBUG { 01:08:46 DEBUG "value": [ 01:08:46 DEBUG "metric": { 01:08:46 DEBUG 01:08:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:08:46 DEBUG 01:08:46 DEBUG }, 01:08:46 DEBUG 01:08:46 DEBUG "value": [ 01:08:46 DEBUG [...] 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG [...] 01:08:46 DEBUG "pod": "ds-idrepo-2" 01:08:46 DEBUG 01:08:46 DEBUG }, 01:08:46 DEBUG 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:08:46 DEBUG "0" 01:08:46 DEBUG }, 01:08:46 DEBUG ] 01:08:46 DEBUG "value": [ 01:08:46 DEBUG } 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG ] 01:08:46 DEBUG "0" 01:08:46 DEBUG } 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG } 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG } 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485326" 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485326" 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG --- http response --- 01:08:46 DEBUG --- http response --- 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG { 01:08:46 DEBUG { 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG "data": { 01:08:46 DEBUG "data": { 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG "result": [ 01:08:46 DEBUG "result": [ 01:08:46 DEBUG { 01:08:46 DEBUG { 01:08:46 DEBUG "metric": { 01:08:46 DEBUG "metric": { 01:08:46 DEBUG "pod": "ds-idrepo-0" 01:08:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:08:46 DEBUG }, 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG [...] 01:08:46 DEBUG [...] 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG "pod": "ds-idrepo-2" 01:08:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:08:46 DEBUG }, 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG "0" 01:08:46 DEBUG "1937.2" 01:08:46 DEBUG ] 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG } 01:08:46 DEBUG ] 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG } 01:08:46 DEBUG } 01:08:46 DEBUG } 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485326" 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG --- http response --- 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG { 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG "data": { 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG "result": [ 01:08:46 DEBUG { 01:08:46 DEBUG "metric": { 01:08:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG [...] 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG "2874361" 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG } 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485326" 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485326" 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG --- http response --- 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG { 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG "data": { 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG "result": [ 01:08:46 DEBUG { 01:08:46 DEBUG "metric": { 01:08:46 DEBUG "instance": "10.142.0.107:9100" 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG [...] 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG "instance": "10.142.0.43:9100" 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG "47294947.53333333" 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG } 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG --- http response --- 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG { 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG "data": { 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG "result": [ 01:08:46 DEBUG { 01:08:46 DEBUG "metric": { 01:08:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG [...] 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG "1937.1" 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG } 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485326" 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485326" 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485326" 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485326" 01:08:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485326" 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG --- http response --- 01:08:46 DEBUG --- http response --- 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG { 01:08:46 DEBUG { 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG --- http response --- 01:08:46 DEBUG "data": { 01:08:46 DEBUG "data": { 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG "result": [ 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 DEBUG "result": [ 01:08:46 DEBUG { 01:08:46 DEBUG { 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG { 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG "metric": { 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG "metric": { 01:08:46 DEBUG "data": { 01:08:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:08:46 DEBUG --- http response --- 01:08:46 DEBUG "pod": "ds-cts-0" 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG }, 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:08:46 DEBUG }, 01:08:46 DEBUG "result": [ 01:08:46 DEBUG "value": [ 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG "value": [ 01:08:46 DEBUG { 01:08:46 DEBUG 01:08:46 DEBUG { 01:08:46 DEBUG 01:08:46 DEBUG "metric": { 01:08:46 DEBUG 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG 01:08:46 DEBUG "pod": "ds-cts-0" 01:08:46 DEBUG 01:08:46 DEBUG "data": { 01:08:46 DEBUG 01:08:46 DEBUG }, 01:08:46 DEBUG [...] 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG [...] 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 01:08:46 DEBUG "result": [ 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG { 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG "metric": { 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:08:46 DEBUG "instance": "10.142.0.107:9100" 01:08:46 DEBUG "pod": "ds-cts-2" 01:08:46 DEBUG [...] 01:08:46 DEBUG }, 01:08:46 DEBUG }, 01:08:46 DEBUG }, 01:08:46 DEBUG 01:08:46 DEBUG "value": [ 01:08:46 DEBUG "value": [ 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG 01:08:46 DEBUG "20.076" 01:08:46 DEBUG 01:08:46 DEBUG "2318.7666666666664" 01:08:46 DEBUG "pod": "ds-cts-2" 01:08:46 DEBUG ] 01:08:46 DEBUG 01:08:46 DEBUG ] 01:08:46 DEBUG }, 01:08:46 DEBUG } 01:08:46 DEBUG [...] 01:08:46 DEBUG } 01:08:46 INFO [http_cmd]: http status code OK 01:08:46 DEBUG "value": [ 01:08:46 DEBUG ] 01:08:46 DEBUG 01:08:46 DEBUG ] 01:08:46 DEBUG --- status code --- 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG } 01:08:46 DEBUG 01:08:46 DEBUG } 01:08:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:08:46 DEBUG "0.00038926" 01:08:46 DEBUG } 01:08:46 DEBUG 01:08:46 DEBUG } 01:08:46 DEBUG --- http response --- 01:08:46 DEBUG ] 01:08:46 DEBUG "instance": "10.142.0.43:9100" 01:08:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:08:46 DEBUG } 01:08:46 DEBUG }, 01:08:46 DEBUG ----- output ----- 01:08:46 DEBUG ] 01:08:46 DEBUG "value": [ 01:08:46 DEBUG { 01:08:46 DEBUG } 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG "status": "success", 01:08:46 DEBUG } 01:08:46 DEBUG "1.2109999999999985" 01:08:46 DEBUG "data": { 01:08:46 DEBUG ] 01:08:46 DEBUG "resultType": "vector", 01:08:46 DEBUG } 01:08:46 DEBUG "result": [ 01:08:46 DEBUG ] 01:08:46 DEBUG { 01:08:46 DEBUG } 01:08:46 DEBUG "metric": { 01:08:46 DEBUG } 01:08:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG [...] 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG 01:08:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:08:46 DEBUG }, 01:08:46 DEBUG "value": [ 01:08:46 DEBUG 1713485326, 01:08:46 DEBUG "1075617792" 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG ] 01:08:46 DEBUG } 01:08:46 DEBUG } 01:08:47 INFO 01:08:47 INFO [loop_until]: kubectl --namespace=xlou top pods 01:08:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:08:47 INFO [loop_until]: OK (rc = 0) 01:08:47 DEBUG --- stdout --- 01:08:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10963m 9398Mi am-6fd554f8f7-hvvkw 8682m 8879Mi am-6fd554f8f7-pmj4b 9015m 9038Mi ds-cts-0 9273m 12353Mi ds-cts-1 9005m 12421Mi ds-cts-2 7962m 12404Mi ds-idrepo-0 4657m 3258Mi ds-idrepo-1 4616m 3126Mi ds-idrepo-2 4678m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 4m 1214Mi lodemon-5d46798758-5h2st 2m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6925m 931Mi 01:08:47 DEBUG --- stderr --- 01:08:47 DEBUG 01:08:47 INFO 01:08:47 INFO [loop_until]: kubectl --namespace=xlou top node 01:08:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:08:47 INFO [loop_until]: OK (rc = 0) 01:08:47 DEBUG --- stdout --- 01:08:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5928Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4659m 15% 6650Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8364m 28% 11608Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3235Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9516m 31% 14819Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 153m 0% 5233Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10887m 36% 12192Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28223m 94% 3133Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10057m 33% 11257Mi 10% gke-xlou-cdm-ds-0f601751-j488 5012m 16% 6094Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9116m 30% 16433Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5303m 17% 6764Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8029m 26% 14319Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7107m 44% 4670Mi 7% 01:08:47 DEBUG --- stderr --- 01:08:47 DEBUG 01:09:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485385" 01:09:45 INFO [http_cmd]: http status code OK 01:09:45 DEBUG --- status code --- 01:09:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:45 DEBUG --- http response --- 01:09:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:45 DEBUG ----- output ----- 01:09:45 DEBUG { 01:09:45 DEBUG "status": "success", 01:09:45 DEBUG "data": { 01:09:45 DEBUG "resultType": "vector", 01:09:45 DEBUG "result": [ 01:09:45 DEBUG { 01:09:45 DEBUG "metric": { 01:09:45 DEBUG "instance": "10.142.0.107:9100" 01:09:45 DEBUG }, 01:09:45 DEBUG "value": [ 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG [...] 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG "instance": "10.142.0.43:9100" 01:09:45 DEBUG }, 01:09:45 DEBUG "value": [ 01:09:45 DEBUG 1713485385, 01:09:45 DEBUG "0.9384333333333292" 01:09:45 DEBUG ] 01:09:45 DEBUG } 01:09:45 DEBUG ] 01:09:45 DEBUG } 01:09:45 DEBUG } 01:09:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485385" 01:09:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485385" 01:09:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485385" 01:09:45 ERROR [http_cmd]: ERROR 01:09:45 ERROR --- status code --- 01:09:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:09:45 ERROR --- http response --- 01:09:45 INFO [http_cmd]: http status code OK 01:09:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:09:45 DEBUG --- status code --- 01:09:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:09:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:45 DEBUG --- http response --- 01:09:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:09:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485385" 01:09:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485385" 01:09:45 INFO [http_cmd]: http status code OK 01:09:45 INFO [http_cmd]: http status code OK 01:09:45 DEBUG --- status code --- 01:09:45 DEBUG --- status code --- 01:09:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:45 DEBUG --- http response --- 01:09:45 DEBUG --- http response --- 01:09:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:45 DEBUG ----- output ----- 01:09:45 DEBUG ----- output ----- 01:09:45 DEBUG { 01:09:45 DEBUG { 01:09:45 DEBUG "status": "success", 01:09:45 DEBUG "status": "success", 01:09:45 DEBUG "data": { 01:09:45 DEBUG "data": { 01:09:45 DEBUG "resultType": "vector", 01:09:45 DEBUG "resultType": "vector", 01:09:45 DEBUG "result": [ 01:09:45 DEBUG "result": [ 01:09:45 DEBUG { 01:09:45 DEBUG { 01:09:45 DEBUG "metric": { 01:09:45 DEBUG "metric": { 01:09:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:09:45 DEBUG "instance": "10.142.0.107:9100" 01:09:45 DEBUG }, 01:09:45 DEBUG }, 01:09:45 DEBUG "value": [ 01:09:45 DEBUG "value": [ 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG [...] 01:09:45 DEBUG 01:09:45 DEBUG [...] 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:09:45 DEBUG 01:09:45 DEBUG }, 01:09:45 DEBUG "instance": "10.142.0.43:9100" 01:09:45 DEBUG "value": [ 01:09:45 DEBUG }, 01:09:45 DEBUG 1713485385, 01:09:45 DEBUG "value": [ 01:09:45 DEBUG "0" 01:09:45 DEBUG 1713485385, 01:09:45 DEBUG ] 01:09:45 DEBUG "333831447552" 01:09:45 DEBUG } 01:09:45 DEBUG ] 01:09:45 DEBUG ] 01:09:45 DEBUG } 01:09:45 DEBUG } 01:09:45 DEBUG ] 01:09:45 DEBUG } 01:09:45 DEBUG } 01:09:45 DEBUG } 01:09:45 INFO [http_cmd]: http status code OK 01:09:45 DEBUG --- status code --- 01:09:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:45 DEBUG --- http response --- 01:09:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:45 DEBUG ----- output ----- 01:09:45 DEBUG { 01:09:45 DEBUG "status": "success", 01:09:45 DEBUG "data": { 01:09:45 DEBUG "resultType": "vector", 01:09:45 DEBUG "result": [ 01:09:45 DEBUG { 01:09:45 DEBUG "metric": { 01:09:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:09:45 DEBUG }, 01:09:45 DEBUG "value": [ 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG [...] 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG "pod": "ds-cts-0" 01:09:45 DEBUG }, 01:09:45 DEBUG "value": [ 01:09:45 DEBUG 1713485385, 01:09:45 DEBUG "43962209.34358367" 01:09:45 DEBUG ] 01:09:45 DEBUG } 01:09:45 DEBUG ] 01:09:45 DEBUG } 01:09:45 DEBUG } 01:09:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485385" 01:09:45 INFO [http_cmd]: http status code OK 01:09:45 DEBUG --- status code --- 01:09:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:45 DEBUG --- http response --- 01:09:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:09:45 DEBUG ----- output ----- 01:09:45 DEBUG { 01:09:45 DEBUG "status": "success", 01:09:45 DEBUG "data": { 01:09:45 DEBUG "resultType": "vector", 01:09:45 DEBUG "result": [ 01:09:45 DEBUG { 01:09:45 DEBUG "metric": { 01:09:45 DEBUG "pod": "ds-idrepo-0" 01:09:45 DEBUG }, 01:09:45 DEBUG "value": [ 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG [...] 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG "pod": "ds-cts-2" 01:09:45 DEBUG }, 01:09:45 DEBUG "value": [ 01:09:45 DEBUG 1713485385, 01:09:45 DEBUG "0" 01:09:45 DEBUG ] 01:09:45 DEBUG } 01:09:45 DEBUG ] 01:09:45 DEBUG } 01:09:45 DEBUG } 01:09:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485385" 01:09:45 INFO [http_cmd]: http status code OK 01:09:45 DEBUG --- status code --- 01:09:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:45 DEBUG --- http response --- 01:09:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:45 DEBUG ----- output ----- 01:09:45 DEBUG { 01:09:45 DEBUG "status": "success", 01:09:45 DEBUG "data": { 01:09:45 DEBUG "resultType": "vector", 01:09:45 DEBUG "result": [ 01:09:45 DEBUG { 01:09:45 DEBUG "metric": { 01:09:45 DEBUG "instance": "10.142.0.107:9100" 01:09:45 DEBUG }, 01:09:45 DEBUG "value": [ 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG [...] 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG "instance": "10.142.0.43:9100" 01:09:45 DEBUG }, 01:09:45 DEBUG "value": [ 01:09:45 DEBUG 1713485385, 01:09:45 DEBUG "47747608.5" 01:09:45 DEBUG ] 01:09:45 DEBUG } 01:09:45 DEBUG ] 01:09:45 DEBUG } 01:09:45 DEBUG } 01:09:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485385" 01:09:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485385" 01:09:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485385" 01:09:45 INFO [http_cmd]: http status code OK 01:09:45 DEBUG --- status code --- 01:09:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:45 DEBUG --- http response --- 01:09:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:45 DEBUG ----- output ----- 01:09:45 DEBUG { 01:09:45 DEBUG "status": "success", 01:09:45 DEBUG "data": { 01:09:45 DEBUG "resultType": "vector", 01:09:45 DEBUG "result": [ 01:09:45 DEBUG { 01:09:45 DEBUG "metric": { 01:09:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:09:45 DEBUG }, 01:09:45 DEBUG "value": [ 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG [...] 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG "pod": "ds-cts-0" 01:09:45 DEBUG }, 01:09:45 DEBUG "value": [ 01:09:45 DEBUG 1713485385, 01:09:45 DEBUG "477227751.6262566" 01:09:45 DEBUG ] 01:09:45 DEBUG } 01:09:45 DEBUG ] 01:09:45 DEBUG } 01:09:45 DEBUG } 01:09:45 INFO [http_cmd]: http status code OK 01:09:45 DEBUG --- status code --- 01:09:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:45 DEBUG --- http response --- 01:09:45 INFO [http_cmd]: http status code OK 01:09:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:09:45 DEBUG ----- output ----- 01:09:45 DEBUG --- status code --- 01:09:45 DEBUG { 01:09:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:45 DEBUG "status": "success", 01:09:45 DEBUG --- http response --- 01:09:45 DEBUG "data": { 01:09:45 DEBUG "resultType": "vector", 01:09:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:45 DEBUG "result": [ 01:09:45 DEBUG ----- output ----- 01:09:45 DEBUG { 01:09:45 DEBUG { 01:09:45 DEBUG "metric": { 01:09:45 DEBUG "status": "success", 01:09:45 DEBUG "pod": "ds-cts-0" 01:09:45 DEBUG "data": { 01:09:45 DEBUG }, 01:09:45 DEBUG "resultType": "vector", 01:09:45 DEBUG "value": [ 01:09:45 DEBUG "result": [ 01:09:45 DEBUG 01:09:45 DEBUG { 01:09:45 DEBUG 01:09:45 DEBUG "metric": { 01:09:45 DEBUG 01:09:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:09:45 DEBUG [...] 01:09:45 DEBUG }, 01:09:45 DEBUG 01:09:45 DEBUG "value": [ 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG 01:09:45 DEBUG "pod": "ds-cts-2" 01:09:45 DEBUG 01:09:45 DEBUG }, 01:09:45 DEBUG [...] 01:09:45 DEBUG "value": [ 01:09:45 DEBUG 01:09:45 DEBUG 1713485385, 01:09:45 DEBUG 01:09:45 DEBUG "0" 01:09:45 DEBUG 01:09:45 DEBUG ] 01:09:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:09:45 DEBUG } 01:09:45 DEBUG }, 01:09:45 DEBUG ] 01:09:45 DEBUG "value": [ 01:09:45 DEBUG } 01:09:45 DEBUG 1713485385, 01:09:45 DEBUG } 01:09:45 DEBUG "1905.8666666666666" 01:09:45 DEBUG ] 01:09:45 DEBUG } 01:09:45 DEBUG ] 01:09:45 DEBUG } 01:09:45 DEBUG } 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485386" 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG --- http response --- 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG { 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG "data": { 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG "result": [ 01:09:46 DEBUG { 01:09:46 DEBUG "metric": { 01:09:46 DEBUG "pod": "ds-idrepo-0" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG [...] 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG "pod": "ds-idrepo-2" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG "0" 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485386" 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485386" 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG --- http response --- 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG { 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG --- http response --- 01:09:46 DEBUG "data": { 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG "result": [ 01:09:46 DEBUG { 01:09:46 DEBUG "metric": { 01:09:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG [...] 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG "pod": "ds-cts-0" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG "9.060137455647503" 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG { 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG "data": { 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG "result": [ 01:09:46 DEBUG { 01:09:46 DEBUG "metric": { 01:09:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG [...] 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG "6.940282153983529" 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485386" 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG --- http response --- 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG { 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG "data": { 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG "result": [ 01:09:46 DEBUG { 01:09:46 DEBUG "metric": { 01:09:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG [...] 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG "0" 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485386" 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485386" 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG --- http response --- 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG { 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG "data": { 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG "result": [ 01:09:46 DEBUG { 01:09:46 DEBUG "metric": { 01:09:46 DEBUG "pod": "ds-idrepo-0" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG [...] 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG "pod": "ds-idrepo-2" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG "0" 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG --- http response --- 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG { 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG "data": { 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG "result": [ 01:09:46 DEBUG { 01:09:46 DEBUG "metric": { 01:09:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG [...] 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG "pod": "ds-cts-0" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG "20866195.150798343" 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485386" 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485386" 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG --- http response --- 01:09:46 DEBUG --- http response --- 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG { 01:09:46 DEBUG { 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG "data": { 01:09:46 DEBUG "data": { 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG "result": [ 01:09:46 DEBUG "result": [ 01:09:46 DEBUG { 01:09:46 DEBUG { 01:09:46 DEBUG "metric": { 01:09:46 DEBUG "metric": { 01:09:46 DEBUG "pod": "ds-idrepo-0" 01:09:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:09:46 DEBUG }, 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG [...] 01:09:46 DEBUG [...] 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG "pod": "ds-idrepo-2" 01:09:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:09:46 DEBUG }, 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG "0" 01:09:46 DEBUG "1905.3" 01:09:46 DEBUG ] 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:46 DEBUG ] 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485386" 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG --- http response --- 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG { 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG "data": { 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG "result": [ 01:09:46 DEBUG { 01:09:46 DEBUG "metric": { 01:09:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG [...] 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG "2989575" 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485386" 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485386" 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG --- http response --- 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG { 01:09:46 DEBUG --- http response --- 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:46 DEBUG "data": { 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG { 01:09:46 DEBUG "result": [ 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG { 01:09:46 DEBUG "data": { 01:09:46 DEBUG "metric": { 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:09:46 DEBUG "result": [ 01:09:46 DEBUG }, 01:09:46 DEBUG { 01:09:46 DEBUG "value": [ 01:09:46 DEBUG "metric": { 01:09:46 DEBUG 01:09:46 DEBUG "instance": "10.142.0.107:9100" 01:09:46 DEBUG 01:09:46 DEBUG }, 01:09:46 DEBUG 01:09:46 DEBUG "value": [ 01:09:46 DEBUG [...] 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG [...] 01:09:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:09:46 DEBUG 01:09:46 DEBUG }, 01:09:46 DEBUG 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG "instance": "10.142.0.43:9100" 01:09:46 DEBUG "1905.7666666666667" 01:09:46 DEBUG }, 01:09:46 DEBUG ] 01:09:46 DEBUG "value": [ 01:09:46 DEBUG } 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG ] 01:09:46 DEBUG "47752915.3" 01:09:46 DEBUG } 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485386" 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485386" 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485386" 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485386" 01:09:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485386" 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG --- http response --- 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG { 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG "data": { 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG "result": [ 01:09:46 DEBUG { 01:09:46 DEBUG "metric": { 01:09:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG [...] 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG "20.076" 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG --- http response --- 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG { 01:09:46 DEBUG --- http response --- 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG "data": { 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:09:46 DEBUG "result": [ 01:09:46 DEBUG ----- output ----- 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG { 01:09:46 DEBUG { 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG "metric": { 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG "pod": "ds-cts-0" 01:09:46 DEBUG "data": { 01:09:46 DEBUG --- http response --- 01:09:46 DEBUG }, 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG "value": [ 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:46 DEBUG "result": [ 01:09:46 DEBUG 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG { 01:09:46 DEBUG 01:09:46 DEBUG { 01:09:46 DEBUG "metric": { 01:09:46 DEBUG 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG "pod": "ds-cts-0" 01:09:46 DEBUG [...] 01:09:46 DEBUG "data": { 01:09:46 DEBUG }, 01:09:46 DEBUG 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 01:09:46 DEBUG "result": [ 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG { 01:09:46 DEBUG 01:09:46 DEBUG "pod": "ds-cts-2" 01:09:46 DEBUG "metric": { 01:09:46 DEBUG 01:09:46 DEBUG }, 01:09:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:09:46 DEBUG [...] 01:09:46 DEBUG "value": [ 01:09:46 DEBUG }, 01:09:46 DEBUG 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 01:09:46 DEBUG "0.00005865333333333333" 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG ] 01:09:46 DEBUG 01:09:46 DEBUG "pod": "ds-cts-2" 01:09:46 DEBUG } 01:09:46 DEBUG 01:09:46 DEBUG }, 01:09:46 INFO [http_cmd]: http status code OK 01:09:46 DEBUG ] 01:09:46 DEBUG [...] 01:09:46 DEBUG "value": [ 01:09:46 DEBUG --- status code --- 01:09:46 DEBUG } 01:09:46 DEBUG 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:09:46 DEBUG } 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG --- http response --- 01:09:46 DEBUG "151.26666666666665" 01:09:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:09:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:09:46 DEBUG ] 01:09:46 DEBUG }, 01:09:46 DEBUG ----- output ----- 01:09:46 DEBUG } 01:09:46 DEBUG "value": [ 01:09:46 DEBUG { 01:09:46 DEBUG ] 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG "status": "success", 01:09:46 DEBUG } 01:09:46 DEBUG "1076961280" 01:09:46 DEBUG "data": { 01:09:46 DEBUG } 01:09:46 DEBUG ] 01:09:46 DEBUG "resultType": "vector", 01:09:46 DEBUG } 01:09:46 DEBUG "result": [ 01:09:46 DEBUG ] 01:09:46 DEBUG { 01:09:46 DEBUG } 01:09:46 DEBUG "metric": { 01:09:46 DEBUG } 01:09:46 DEBUG "instance": "10.142.0.107:9100" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG [...] 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG 01:09:46 DEBUG "instance": "10.142.0.43:9100" 01:09:46 DEBUG }, 01:09:46 DEBUG "value": [ 01:09:46 DEBUG 1713485386, 01:09:46 DEBUG "0.9650000000000012" 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG ] 01:09:46 DEBUG } 01:09:46 DEBUG } 01:09:47 INFO 01:09:47 INFO [loop_until]: kubectl --namespace=xlou top pods 01:09:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:09:47 INFO [loop_until]: OK (rc = 0) 01:09:47 DEBUG --- stdout --- 01:09:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10796m 9400Mi am-6fd554f8f7-hvvkw 8425m 8880Mi am-6fd554f8f7-pmj4b 9483m 9039Mi ds-cts-0 8826m 12313Mi ds-cts-1 7651m 12391Mi ds-cts-2 8483m 12335Mi ds-idrepo-0 4679m 3258Mi ds-idrepo-1 4586m 3125Mi ds-idrepo-2 4629m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1171Mi idm-b85f9d978-wnhxb 4m 1214Mi lodemon-5d46798758-5h2st 10m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7024m 932Mi 01:09:47 DEBUG --- stderr --- 01:09:47 DEBUG 01:09:47 INFO 01:09:47 INFO [loop_until]: kubectl --namespace=xlou top node 01:09:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:09:47 INFO [loop_until]: OK (rc = 0) 01:09:47 DEBUG --- stdout --- 01:09:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5912Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4616m 15% 6663Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8479m 28% 11600Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 82m 0% 3234Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8630m 28% 14789Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5206Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10834m 36% 12198Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27941m 93% 3130Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10059m 33% 11253Mi 10% gke-xlou-cdm-ds-0f601751-j488 5006m 16% 6096Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8140m 27% 16364Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5268m 17% 6758Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8595m 28% 14256Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7088m 44% 4668Mi 7% 01:09:47 DEBUG --- stderr --- 01:09:47 DEBUG 01:10:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485445" 01:10:45 INFO [http_cmd]: http status code OK 01:10:45 DEBUG --- status code --- 01:10:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:45 DEBUG --- http response --- 01:10:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:45 DEBUG ----- output ----- 01:10:45 DEBUG { 01:10:45 DEBUG "status": "success", 01:10:45 DEBUG "data": { 01:10:45 DEBUG "resultType": "vector", 01:10:45 DEBUG "result": [ 01:10:45 DEBUG { 01:10:45 DEBUG "metric": { 01:10:45 DEBUG "instance": "10.142.0.107:9100" 01:10:45 DEBUG }, 01:10:45 DEBUG "value": [ 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG [...] 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG "instance": "10.142.0.43:9100" 01:10:45 DEBUG }, 01:10:45 DEBUG "value": [ 01:10:45 DEBUG 1713485445, 01:10:45 DEBUG "0.9359666666666678" 01:10:45 DEBUG ] 01:10:45 DEBUG } 01:10:45 DEBUG ] 01:10:45 DEBUG } 01:10:45 DEBUG } 01:10:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485445" 01:10:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485445" 01:10:45 ERROR [http_cmd]: ERROR 01:10:45 ERROR --- status code --- 01:10:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:10:45 ERROR --- http response --- 01:10:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:10:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:10:45 INFO [http_cmd]: http status code OK 01:10:45 DEBUG --- status code --- 01:10:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:45 DEBUG --- http response --- 01:10:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:10:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485445" 01:10:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485445" 01:10:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485445" 01:10:45 INFO [http_cmd]: http status code OK 01:10:45 DEBUG --- status code --- 01:10:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:45 DEBUG --- http response --- 01:10:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:45 DEBUG ----- output ----- 01:10:45 DEBUG { 01:10:45 DEBUG "status": "success", 01:10:45 DEBUG "data": { 01:10:45 DEBUG "resultType": "vector", 01:10:45 DEBUG "result": [ 01:10:45 DEBUG { 01:10:45 DEBUG "metric": { 01:10:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:10:45 DEBUG }, 01:10:45 DEBUG "value": [ 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG [...] 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:10:45 DEBUG }, 01:10:45 DEBUG "value": [ 01:10:45 DEBUG 1713485445, 01:10:45 DEBUG "0" 01:10:45 DEBUG ] 01:10:45 DEBUG } 01:10:45 DEBUG ] 01:10:45 DEBUG } 01:10:45 DEBUG } 01:10:45 INFO [http_cmd]: http status code OK 01:10:45 DEBUG --- status code --- 01:10:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:45 DEBUG --- http response --- 01:10:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:45 DEBUG ----- output ----- 01:10:45 DEBUG { 01:10:45 DEBUG "status": "success", 01:10:45 DEBUG "data": { 01:10:45 DEBUG "resultType": "vector", 01:10:45 DEBUG "result": [ 01:10:45 DEBUG { 01:10:45 DEBUG "metric": { 01:10:45 DEBUG "instance": "10.142.0.107:9100" 01:10:45 DEBUG }, 01:10:45 DEBUG "value": [ 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG [...] 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG "instance": "10.142.0.43:9100" 01:10:45 DEBUG }, 01:10:45 DEBUG "value": [ 01:10:45 DEBUG 1713485445, 01:10:45 DEBUG "336258026496" 01:10:45 DEBUG ] 01:10:45 DEBUG } 01:10:45 DEBUG ] 01:10:45 DEBUG } 01:10:45 DEBUG } 01:10:45 INFO [http_cmd]: http status code OK 01:10:45 DEBUG --- status code --- 01:10:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:45 DEBUG --- http response --- 01:10:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:10:45 DEBUG ----- output ----- 01:10:45 DEBUG { 01:10:45 DEBUG "status": "success", 01:10:45 DEBUG "data": { 01:10:45 DEBUG "resultType": "vector", 01:10:45 DEBUG "result": [ 01:10:45 DEBUG { 01:10:45 DEBUG "metric": { 01:10:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:10:45 DEBUG }, 01:10:45 DEBUG "value": [ 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG [...] 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:10:45 DEBUG }, 01:10:45 DEBUG "value": [ 01:10:45 DEBUG 1713485445, 01:10:45 DEBUG "0" 01:10:45 DEBUG ] 01:10:45 DEBUG } 01:10:45 DEBUG ] 01:10:45 DEBUG } 01:10:45 DEBUG } 01:10:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485445" 01:10:45 INFO [http_cmd]: http status code OK 01:10:45 DEBUG --- status code --- 01:10:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:45 DEBUG --- http response --- 01:10:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:10:45 DEBUG ----- output ----- 01:10:45 DEBUG { 01:10:45 DEBUG "status": "success", 01:10:45 DEBUG "data": { 01:10:45 DEBUG "resultType": "vector", 01:10:45 DEBUG "result": [ 01:10:45 DEBUG { 01:10:45 DEBUG "metric": { 01:10:45 DEBUG "pod": "ds-idrepo-0" 01:10:45 DEBUG }, 01:10:45 DEBUG "value": [ 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG [...] 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG 01:10:45 DEBUG "pod": "ds-cts-2" 01:10:45 DEBUG }, 01:10:45 DEBUG "value": [ 01:10:45 DEBUG 1713485445, 01:10:45 DEBUG "0" 01:10:45 DEBUG ] 01:10:45 DEBUG } 01:10:45 DEBUG ] 01:10:45 DEBUG } 01:10:45 DEBUG } 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485446" 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "instance": "10.142.0.107:9100" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "instance": "10.142.0.43:9100" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "47677929.6" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG { 01:10:46 DEBUG "result": [ 01:10:46 DEBUG "metric": { 01:10:46 DEBUG { 01:10:46 DEBUG "pod": "ds-idrepo-0" 01:10:46 DEBUG "metric": { 01:10:46 DEBUG }, 01:10:46 DEBUG "pod": "ds-cts-0" 01:10:46 DEBUG "value": [ 01:10:46 DEBUG }, 01:10:46 DEBUG 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG 01:10:46 DEBUG "pod": "ds-idrepo-2" 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG 01:10:46 DEBUG }, 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG "pod": "ds-cts-2" 01:10:46 DEBUG "value": [ 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:10:46 DEBUG }, 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG "value": [ 01:10:46 DEBUG "0" 01:10:46 DEBUG { 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG ] 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "0" 01:10:46 DEBUG } 01:10:46 DEBUG "data": { 01:10:46 DEBUG ] 01:10:46 DEBUG ] 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 DEBUG "result": [ 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG { 01:10:46 DEBUG } 01:10:46 DEBUG "metric": { 01:10:46 DEBUG } 01:10:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "6177.185945772502" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "1914.5333333333333" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "0" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485446" 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:10:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:10:46 DEBUG }, 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:10:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:10:46 DEBUG }, 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "6.897678652943805" 01:10:46 DEBUG "6.918475827054431" 01:10:46 DEBUG ] 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "pod": "ds-idrepo-0" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "pod": "ds-idrepo-2" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "0" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "0" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "pod": "ds-idrepo-0" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "pod": "ds-idrepo-2" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "0" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "1913.6333333333332" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "3104745" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485446" 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "1914.5" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "instance": "10.142.0.107:9100" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "instance": "10.142.0.43:9100" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "47679851.2" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "pod": "ds-cts-0" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "pod": "ds-cts-2" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "0.000033333333333333335" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485446" 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485446" 01:10:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485446" 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "20.076" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "pod": "ds-cts-0" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "pod": "ds-cts-2" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "2620.233333333333" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "instance": "10.142.0.107:9100" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "instance": "10.142.0.43:9100" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "1.030333333333331" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:46 INFO [http_cmd]: http status code OK 01:10:46 DEBUG --- status code --- 01:10:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:10:46 DEBUG --- http response --- 01:10:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:10:46 DEBUG ----- output ----- 01:10:46 DEBUG { 01:10:46 DEBUG "status": "success", 01:10:46 DEBUG "data": { 01:10:46 DEBUG "resultType": "vector", 01:10:46 DEBUG "result": [ 01:10:46 DEBUG { 01:10:46 DEBUG "metric": { 01:10:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG [...] 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG 01:10:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:10:46 DEBUG }, 01:10:46 DEBUG "value": [ 01:10:46 DEBUG 1713485446, 01:10:46 DEBUG "1077911552" 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG ] 01:10:46 DEBUG } 01:10:46 DEBUG } 01:10:47 INFO 01:10:47 INFO [loop_until]: kubectl --namespace=xlou top pods 01:10:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:10:47 INFO [loop_until]: OK (rc = 0) 01:10:47 DEBUG --- stdout --- 01:10:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10424m 9403Mi am-6fd554f8f7-hvvkw 8665m 8885Mi am-6fd554f8f7-pmj4b 9397m 9063Mi ds-cts-0 8936m 12366Mi ds-cts-1 9613m 12398Mi ds-cts-2 8740m 12402Mi ds-idrepo-0 4740m 3258Mi ds-idrepo-1 4793m 3126Mi ds-idrepo-2 4574m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 4m 1215Mi lodemon-5d46798758-5h2st 1m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6912m 935Mi 01:10:47 DEBUG --- stderr --- 01:10:47 DEBUG 01:10:47 INFO 01:10:47 INFO [loop_until]: kubectl --namespace=xlou top node 01:10:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:10:47 INFO [loop_until]: OK (rc = 0) 01:10:47 DEBUG --- stdout --- 01:10:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 5921Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4646m 15% 6660Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8676m 29% 11611Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3248Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9376m 31% 14864Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 137m 0% 5227Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10567m 35% 12197Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 27185m 91% 3136Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10045m 33% 11275Mi 10% gke-xlou-cdm-ds-0f601751-j488 5003m 16% 6104Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 9060m 30% 16390Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5221m 17% 6753Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8840m 29% 14317Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7089m 44% 4669Mi 7% 01:10:47 DEBUG --- stderr --- 01:10:47 DEBUG 01:11:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485505" 01:11:45 INFO [http_cmd]: http status code OK 01:11:45 DEBUG --- status code --- 01:11:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:45 DEBUG --- http response --- 01:11:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:45 DEBUG ----- output ----- 01:11:45 DEBUG { 01:11:45 DEBUG "status": "success", 01:11:45 DEBUG "data": { 01:11:45 DEBUG "resultType": "vector", 01:11:45 DEBUG "result": [ 01:11:45 DEBUG { 01:11:45 DEBUG "metric": { 01:11:45 DEBUG "instance": "10.142.0.107:9100" 01:11:45 DEBUG }, 01:11:45 DEBUG "value": [ 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG [...] 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG "instance": "10.142.0.43:9100" 01:11:45 DEBUG }, 01:11:45 DEBUG "value": [ 01:11:45 DEBUG 1713485505, 01:11:45 DEBUG "0.9694999999999956" 01:11:45 DEBUG ] 01:11:45 DEBUG } 01:11:45 DEBUG ] 01:11:45 DEBUG } 01:11:45 DEBUG } 01:11:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485505" 01:11:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485505" 01:11:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485505" 01:11:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485505" 01:11:45 INFO [http_cmd]: http status code OK 01:11:45 ERROR [http_cmd]: ERROR 01:11:45 DEBUG --- status code --- 01:11:45 ERROR --- status code --- 01:11:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:11:45 DEBUG --- http response --- 01:11:45 ERROR --- http response --- 01:11:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:11:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:11:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:11:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485505" 01:11:45 INFO [http_cmd]: http status code OK 01:11:45 DEBUG --- status code --- 01:11:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:45 DEBUG --- http response --- 01:11:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:45 DEBUG ----- output ----- 01:11:45 DEBUG { 01:11:45 DEBUG "status": "success", 01:11:45 DEBUG "data": { 01:11:45 DEBUG "resultType": "vector", 01:11:45 DEBUG "result": [ 01:11:45 DEBUG { 01:11:45 DEBUG "metric": { 01:11:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:11:45 DEBUG }, 01:11:45 DEBUG "value": [ 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG [...] 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:11:45 DEBUG }, 01:11:45 DEBUG "value": [ 01:11:45 DEBUG 1713485505, 01:11:45 DEBUG "0" 01:11:45 DEBUG ] 01:11:45 DEBUG } 01:11:45 DEBUG ] 01:11:45 DEBUG } 01:11:45 DEBUG } 01:11:45 INFO [http_cmd]: http status code OK 01:11:45 DEBUG --- status code --- 01:11:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:45 DEBUG --- http response --- 01:11:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:45 DEBUG ----- output ----- 01:11:45 DEBUG { 01:11:45 DEBUG "status": "success", 01:11:45 DEBUG "data": { 01:11:45 DEBUG "resultType": "vector", 01:11:45 DEBUG "result": [ 01:11:45 DEBUG { 01:11:45 DEBUG "metric": { 01:11:45 DEBUG "instance": "10.142.0.107:9100" 01:11:45 DEBUG }, 01:11:45 DEBUG "value": [ 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG [...] 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG "instance": "10.142.0.43:9100" 01:11:45 DEBUG }, 01:11:45 DEBUG "value": [ 01:11:45 DEBUG 1713485505, 01:11:45 DEBUG "338688961536" 01:11:45 DEBUG ] 01:11:45 DEBUG } 01:11:45 DEBUG ] 01:11:45 DEBUG } 01:11:45 DEBUG } 01:11:45 INFO [http_cmd]: http status code OK 01:11:45 DEBUG --- status code --- 01:11:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:45 DEBUG --- http response --- 01:11:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:11:45 DEBUG ----- output ----- 01:11:45 DEBUG { 01:11:45 DEBUG "status": "success", 01:11:45 DEBUG "data": { 01:11:45 DEBUG "resultType": "vector", 01:11:45 DEBUG "result": [ 01:11:45 DEBUG { 01:11:45 DEBUG "metric": { 01:11:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:11:45 DEBUG }, 01:11:45 DEBUG "value": [ 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG [...] 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG 01:11:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:11:45 DEBUG }, 01:11:45 DEBUG "value": [ 01:11:45 DEBUG 1713485505, 01:11:45 DEBUG "0" 01:11:45 DEBUG ] 01:11:45 DEBUG } 01:11:45 DEBUG ] 01:11:45 DEBUG } 01:11:45 DEBUG } 01:11:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485505" 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG { 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG "data": { 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG "result": [ 01:11:46 DEBUG { 01:11:46 DEBUG "metric": { 01:11:46 DEBUG "pod": "ds-idrepo-0" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG [...] 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "pod": "ds-cts-2" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 1713485505, 01:11:46 DEBUG "0" 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485506" 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG { 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG "data": { 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG "result": [ 01:11:46 DEBUG { 01:11:46 DEBUG "metric": { 01:11:46 DEBUG "instance": "10.142.0.107:9100" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG [...] 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "instance": "10.142.0.43:9100" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG "47199134.266666666" 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485506" 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485506" 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485506" 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485506" 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG { 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG "data": { 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG "result": [ 01:11:46 DEBUG { 01:11:46 DEBUG "metric": { 01:11:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG [...] 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG "43015.54234466935" 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG ----- output ----- 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 DEBUG { 01:11:46 DEBUG { 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG "data": { 01:11:46 DEBUG "data": { 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:46 DEBUG "result": [ 01:11:46 DEBUG "result": [ 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG { 01:11:46 DEBUG { 01:11:46 DEBUG { 01:11:46 DEBUG "metric": { 01:11:46 DEBUG "metric": { 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG "pod": "ds-cts-0" 01:11:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:11:46 DEBUG "data": { 01:11:46 DEBUG }, 01:11:46 DEBUG }, 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG "value": [ 01:11:46 DEBUG "value": [ 01:11:46 DEBUG "result": [ 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG { 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "metric": { 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "pod": "ds-idrepo-0" 01:11:46 DEBUG [...] 01:11:46 DEBUG [...] 01:11:46 DEBUG }, 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "pod": "ds-cts-2" 01:11:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:11:46 DEBUG 01:11:46 DEBUG }, 01:11:46 DEBUG }, 01:11:46 DEBUG [...] 01:11:46 DEBUG "value": [ 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG 01:11:46 DEBUG "0" 01:11:46 DEBUG "1928.7" 01:11:46 DEBUG 01:11:46 DEBUG ] 01:11:46 DEBUG ] 01:11:46 DEBUG "pod": "ds-idrepo-2" 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 DEBUG }, 01:11:46 DEBUG ] 01:11:46 DEBUG ] 01:11:46 DEBUG "value": [ 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 DEBUG "0" 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485506" 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG { 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG "data": { 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG "result": [ 01:11:46 DEBUG { 01:11:46 DEBUG "metric": { 01:11:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG [...] 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG "0" 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485506" 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485506" 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 DEBUG --- status code --- 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:46 DEBUG { 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG { 01:11:46 DEBUG "data": { 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG "data": { 01:11:46 DEBUG "result": [ 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG { 01:11:46 DEBUG "result": [ 01:11:46 DEBUG "metric": { 01:11:46 DEBUG { 01:11:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:11:46 DEBUG "metric": { 01:11:46 DEBUG }, 01:11:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:11:46 DEBUG "value": [ 01:11:46 DEBUG }, 01:11:46 DEBUG 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG [...] 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG [...] 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:11:46 DEBUG 01:11:46 DEBUG }, 01:11:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:11:46 DEBUG "value": [ 01:11:46 DEBUG }, 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG "13.845314330573045" 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG ] 01:11:46 DEBUG "6.9560376219844855" 01:11:46 DEBUG } 01:11:46 DEBUG ] 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485506" 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485506" 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG { 01:11:46 DEBUG { 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG "data": { 01:11:46 DEBUG "data": { 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG "result": [ 01:11:46 DEBUG "result": [ 01:11:46 DEBUG { 01:11:46 DEBUG { 01:11:46 DEBUG "metric": { 01:11:46 DEBUG "metric": { 01:11:46 DEBUG "pod": "ds-idrepo-0" 01:11:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:11:46 DEBUG }, 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG [...] 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG [...] 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "pod": "ds-idrepo-2" 01:11:46 DEBUG 01:11:46 DEBUG }, 01:11:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:11:46 DEBUG "value": [ 01:11:46 DEBUG }, 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG "0" 01:11:46 DEBUG "0" 01:11:46 DEBUG ] 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 DEBUG ] 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485506" 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485506" 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG { 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG "data": { 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG "result": [ 01:11:46 DEBUG { 01:11:46 DEBUG "metric": { 01:11:46 DEBUG "pod": "ds-idrepo-0" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG [...] 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "pod": "ds-idrepo-2" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG "0" 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG { 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG "data": { 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG "result": [ 01:11:46 DEBUG { 01:11:46 DEBUG "metric": { 01:11:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG [...] 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG "1928.8333333333333" 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485506" 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485506" 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG { 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG "data": { 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG "result": [ 01:11:46 DEBUG { 01:11:46 DEBUG "metric": { 01:11:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG [...] 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG "3219795" 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG { 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG "data": { 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG "result": [ 01:11:46 DEBUG { 01:11:46 DEBUG "metric": { 01:11:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG [...] 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG "1928.7666666666667" 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485506" 01:11:46 INFO [http_cmd]: http status code OK 01:11:46 DEBUG --- status code --- 01:11:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:46 DEBUG --- http response --- 01:11:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:46 DEBUG ----- output ----- 01:11:46 DEBUG { 01:11:46 DEBUG "status": "success", 01:11:46 DEBUG "data": { 01:11:46 DEBUG "resultType": "vector", 01:11:46 DEBUG "result": [ 01:11:46 DEBUG { 01:11:46 DEBUG "metric": { 01:11:46 DEBUG "instance": "10.142.0.107:9100" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG [...] 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG 01:11:46 DEBUG "instance": "10.142.0.43:9100" 01:11:46 DEBUG }, 01:11:46 DEBUG "value": [ 01:11:46 DEBUG 1713485506, 01:11:46 DEBUG "47204340.266666666" 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG ] 01:11:46 DEBUG } 01:11:46 DEBUG } 01:11:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485507" 01:11:47 INFO [http_cmd]: http status code OK 01:11:47 DEBUG --- status code --- 01:11:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:47 DEBUG --- http response --- 01:11:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:11:47 DEBUG ----- output ----- 01:11:47 DEBUG { 01:11:47 DEBUG "status": "success", 01:11:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485507" 01:11:47 DEBUG "data": { 01:11:47 DEBUG "resultType": "vector", 01:11:47 DEBUG "result": [ 01:11:47 DEBUG { 01:11:47 DEBUG "metric": { 01:11:47 DEBUG "pod": "ds-cts-0" 01:11:47 DEBUG }, 01:11:47 DEBUG "value": [ 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG [...] 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG "pod": "ds-cts-2" 01:11:47 DEBUG }, 01:11:47 DEBUG "value": [ 01:11:47 DEBUG 1713485507, 01:11:47 DEBUG "0" 01:11:47 DEBUG ] 01:11:47 DEBUG } 01:11:47 DEBUG ] 01:11:47 DEBUG } 01:11:47 DEBUG } 01:11:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485507" 01:11:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485507" 01:11:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485507" 01:11:47 INFO [http_cmd]: http status code OK 01:11:47 DEBUG --- status code --- 01:11:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:47 DEBUG --- http response --- 01:11:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:11:47 DEBUG ----- output ----- 01:11:47 DEBUG { 01:11:47 DEBUG "status": "success", 01:11:47 DEBUG "data": { 01:11:47 DEBUG "resultType": "vector", 01:11:47 DEBUG "result": [ 01:11:47 DEBUG { 01:11:47 DEBUG "metric": { 01:11:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:11:47 DEBUG }, 01:11:47 DEBUG "value": [ 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG [...] 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:11:47 DEBUG }, 01:11:47 DEBUG "value": [ 01:11:47 DEBUG 1713485507, 01:11:47 DEBUG "20.076" 01:11:47 DEBUG ] 01:11:47 DEBUG } 01:11:47 DEBUG ] 01:11:47 DEBUG } 01:11:47 DEBUG } 01:11:47 INFO [http_cmd]: http status code OK 01:11:47 DEBUG --- status code --- 01:11:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:47 DEBUG --- http response --- 01:11:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:11:47 DEBUG ----- output ----- 01:11:47 DEBUG { 01:11:47 DEBUG "status": "success", 01:11:47 DEBUG "data": { 01:11:47 DEBUG "resultType": "vector", 01:11:47 DEBUG "result": [ 01:11:47 DEBUG { 01:11:47 DEBUG "metric": { 01:11:47 DEBUG "pod": "ds-cts-0" 01:11:47 DEBUG }, 01:11:47 DEBUG "value": [ 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG [...] 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG "pod": "ds-cts-2" 01:11:47 DEBUG }, 01:11:47 DEBUG "value": [ 01:11:47 DEBUG 1713485507, 01:11:47 DEBUG "3344.7666666666664" 01:11:47 DEBUG ] 01:11:47 DEBUG } 01:11:47 DEBUG ] 01:11:47 DEBUG } 01:11:47 DEBUG } 01:11:47 INFO [http_cmd]: http status code OK 01:11:47 DEBUG --- status code --- 01:11:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:47 DEBUG --- http response --- 01:11:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:47 DEBUG ----- output ----- 01:11:47 DEBUG { 01:11:47 DEBUG "status": "success", 01:11:47 DEBUG "data": { 01:11:47 DEBUG "resultType": "vector", 01:11:47 DEBUG "result": [ 01:11:47 DEBUG { 01:11:47 DEBUG "metric": { 01:11:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:11:47 DEBUG }, 01:11:47 DEBUG "value": [ 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG [...] 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:11:47 DEBUG }, 01:11:47 DEBUG "value": [ 01:11:47 DEBUG 1713485507, 01:11:47 DEBUG "1081110528" 01:11:47 DEBUG ] 01:11:47 DEBUG } 01:11:47 DEBUG ] 01:11:47 DEBUG } 01:11:47 DEBUG } 01:11:47 INFO [http_cmd]: http status code OK 01:11:47 DEBUG --- status code --- 01:11:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:11:47 DEBUG --- http response --- 01:11:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:11:47 DEBUG ----- output ----- 01:11:47 DEBUG { 01:11:47 DEBUG "status": "success", 01:11:47 DEBUG "data": { 01:11:47 DEBUG "resultType": "vector", 01:11:47 DEBUG "result": [ 01:11:47 DEBUG { 01:11:47 DEBUG "metric": { 01:11:47 DEBUG "instance": "10.142.0.107:9100" 01:11:47 DEBUG }, 01:11:47 DEBUG "value": [ 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG [...] 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG 01:11:47 DEBUG "instance": "10.142.0.43:9100" 01:11:47 DEBUG }, 01:11:47 DEBUG "value": [ 01:11:47 DEBUG 1713485507, 01:11:47 DEBUG "1.249" 01:11:47 DEBUG ] 01:11:47 DEBUG } 01:11:47 DEBUG ] 01:11:47 DEBUG } 01:11:47 DEBUG } 01:11:47 INFO 01:11:47 INFO [loop_until]: kubectl --namespace=xlou top pods 01:11:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:11:47 INFO [loop_until]: OK (rc = 0) 01:11:47 DEBUG --- stdout --- 01:11:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10873m 9404Mi am-6fd554f8f7-hvvkw 8613m 8885Mi am-6fd554f8f7-pmj4b 9368m 9066Mi ds-cts-0 9135m 12374Mi ds-cts-1 8033m 12351Mi ds-cts-2 8440m 12455Mi ds-idrepo-0 4701m 3257Mi ds-idrepo-1 4710m 3126Mi ds-idrepo-2 4560m 3040Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 9m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6947m 936Mi 01:11:47 DEBUG --- stderr --- 01:11:47 DEBUG 01:11:47 INFO 01:11:47 INFO [loop_until]: kubectl --namespace=xlou top node 01:11:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:11:47 INFO [loop_until]: OK (rc = 0) 01:11:47 DEBUG --- stdout --- 01:11:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5919Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4634m 15% 6661Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8428m 28% 11611Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 81m 0% 3243Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9016m 30% 14887Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5209Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10946m 36% 12194Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28171m 94% 3141Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10196m 34% 11290Mi 10% gke-xlou-cdm-ds-0f601751-j488 5045m 16% 6095Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8103m 27% 16377Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5231m 17% 6749Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8381m 28% 14379Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7123m 44% 4666Mi 7% 01:11:47 DEBUG --- stderr --- 01:11:47 DEBUG 01:12:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485565" 01:12:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485565" 01:12:45 ERROR [http_cmd]: ERROR 01:12:45 ERROR --- status code --- 01:12:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:12:45 ERROR --- http response --- 01:12:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:12:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:12:45 INFO [http_cmd]: http status code OK 01:12:45 DEBUG --- status code --- 01:12:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:45 DEBUG --- http response --- 01:12:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:45 DEBUG ----- output ----- 01:12:45 DEBUG { 01:12:45 DEBUG "status": "success", 01:12:45 DEBUG "data": { 01:12:45 DEBUG "resultType": "vector", 01:12:45 DEBUG "result": [ 01:12:45 DEBUG { 01:12:45 DEBUG "metric": { 01:12:45 DEBUG "instance": "10.142.0.107:9100" 01:12:45 DEBUG }, 01:12:45 DEBUG "value": [ 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG [...] 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG "instance": "10.142.0.43:9100" 01:12:45 DEBUG }, 01:12:45 DEBUG "value": [ 01:12:45 DEBUG 1713485565, 01:12:45 DEBUG "0.934166666666664" 01:12:45 DEBUG ] 01:12:45 DEBUG } 01:12:45 DEBUG ] 01:12:45 DEBUG } 01:12:45 DEBUG } 01:12:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485565" 01:12:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485565" 01:12:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485565" 01:12:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485565" 01:12:45 INFO [http_cmd]: http status code OK 01:12:45 DEBUG --- status code --- 01:12:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:45 DEBUG --- http response --- 01:12:45 INFO [http_cmd]: http status code OK 01:12:45 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:12:45 DEBUG --- status code --- 01:12:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:45 DEBUG --- http response --- 01:12:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:45 DEBUG ----- output ----- 01:12:45 DEBUG { 01:12:45 DEBUG "status": "success", 01:12:45 DEBUG "data": { 01:12:45 DEBUG "resultType": "vector", 01:12:45 DEBUG "result": [ 01:12:45 DEBUG { 01:12:45 DEBUG "metric": { 01:12:45 DEBUG "instance": "10.142.0.107:9100" 01:12:45 DEBUG }, 01:12:45 DEBUG "value": [ 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG [...] 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG "instance": "10.142.0.43:9100" 01:12:45 DEBUG }, 01:12:45 DEBUG "value": [ 01:12:45 DEBUG 1713485565, 01:12:45 DEBUG "341123752960" 01:12:45 DEBUG ] 01:12:45 DEBUG } 01:12:45 DEBUG ] 01:12:45 DEBUG } 01:12:45 DEBUG } 01:12:45 INFO [http_cmd]: http status code OK 01:12:45 DEBUG --- status code --- 01:12:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:45 DEBUG --- http response --- 01:12:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:45 DEBUG ----- output ----- 01:12:45 DEBUG { 01:12:45 DEBUG "status": "success", 01:12:45 DEBUG "data": { 01:12:45 DEBUG "resultType": "vector", 01:12:45 DEBUG "result": [ 01:12:45 DEBUG { 01:12:45 DEBUG "metric": { 01:12:45 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:12:45 DEBUG }, 01:12:45 DEBUG "value": [ 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG [...] 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:12:45 DEBUG }, 01:12:45 DEBUG "value": [ 01:12:45 DEBUG 1713485565, 01:12:45 DEBUG "0" 01:12:45 DEBUG ] 01:12:45 DEBUG } 01:12:45 DEBUG ] 01:12:45 DEBUG } 01:12:45 DEBUG } 01:12:45 INFO [http_cmd]: http status code OK 01:12:45 DEBUG --- status code --- 01:12:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:45 DEBUG --- http response --- 01:12:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:45 DEBUG ----- output ----- 01:12:45 DEBUG { 01:12:45 DEBUG "status": "success", 01:12:45 DEBUG "data": { 01:12:45 DEBUG "resultType": "vector", 01:12:45 DEBUG "result": [ 01:12:45 DEBUG { 01:12:45 DEBUG "metric": { 01:12:45 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:12:45 DEBUG }, 01:12:45 DEBUG "value": [ 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG [...] 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG 01:12:45 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:12:45 DEBUG }, 01:12:45 DEBUG "value": [ 01:12:45 DEBUG 1713485565, 01:12:45 DEBUG "0" 01:12:45 DEBUG ] 01:12:45 DEBUG } 01:12:45 DEBUG ] 01:12:45 DEBUG } 01:12:45 DEBUG } 01:12:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485566" 01:12:46 INFO [http_cmd]: http status code OK 01:12:46 DEBUG --- status code --- 01:12:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:46 DEBUG --- http response --- 01:12:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:12:46 DEBUG ----- output ----- 01:12:46 DEBUG { 01:12:46 DEBUG "status": "success", 01:12:46 DEBUG "data": { 01:12:46 DEBUG "resultType": "vector", 01:12:46 DEBUG "result": [ 01:12:46 DEBUG { 01:12:46 DEBUG "metric": { 01:12:46 DEBUG "pod": "ds-idrepo-0" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG [...] 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG "pod": "ds-cts-2" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 1713485566, 01:12:46 DEBUG "0" 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG } 01:12:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485566" 01:12:46 INFO [http_cmd]: http status code OK 01:12:46 DEBUG --- status code --- 01:12:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:46 DEBUG --- http response --- 01:12:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:46 DEBUG ----- output ----- 01:12:46 DEBUG { 01:12:46 DEBUG "status": "success", 01:12:46 DEBUG "data": { 01:12:46 DEBUG "resultType": "vector", 01:12:46 DEBUG "result": [ 01:12:46 DEBUG { 01:12:46 DEBUG "metric": { 01:12:46 DEBUG "instance": "10.142.0.107:9100" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG [...] 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG "instance": "10.142.0.43:9100" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 1713485566, 01:12:46 DEBUG "47228700.8" 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG } 01:12:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485566" 01:12:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485566" 01:12:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485566" 01:12:46 INFO [http_cmd]: http status code OK 01:12:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485566" 01:12:46 DEBUG --- status code --- 01:12:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:46 DEBUG --- http response --- 01:12:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:12:46 DEBUG ----- output ----- 01:12:46 DEBUG { 01:12:46 DEBUG "status": "success", 01:12:46 DEBUG "data": { 01:12:46 DEBUG "resultType": "vector", 01:12:46 DEBUG "result": [ 01:12:46 DEBUG { 01:12:46 DEBUG "metric": { 01:12:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG [...] 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG "pod": "ds-cts-2" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 1713485566, 01:12:46 DEBUG "0" 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG } 01:12:46 INFO [http_cmd]: http status code OK 01:12:46 DEBUG --- status code --- 01:12:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:46 DEBUG --- http response --- 01:12:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:12:46 DEBUG ----- output ----- 01:12:46 DEBUG { 01:12:46 DEBUG "status": "success", 01:12:46 DEBUG "data": { 01:12:46 DEBUG "resultType": "vector", 01:12:46 DEBUG "result": [ 01:12:46 DEBUG { 01:12:46 DEBUG "metric": { 01:12:46 DEBUG "pod": "ds-cts-0" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG [...] 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG "pod": "ds-cts-2" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 1713485566, 01:12:46 DEBUG "0" 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG } 01:12:46 INFO [http_cmd]: http status code OK 01:12:46 DEBUG --- status code --- 01:12:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:46 DEBUG --- http response --- 01:12:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:46 DEBUG ----- output ----- 01:12:46 DEBUG { 01:12:46 DEBUG "status": "success", 01:12:46 DEBUG "data": { 01:12:46 DEBUG "resultType": "vector", 01:12:46 DEBUG "result": [ 01:12:46 DEBUG { 01:12:46 DEBUG "metric": { 01:12:46 DEBUG "pod": "ds-idrepo-0" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG [...] 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG "pod": "ds-idrepo-2" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 1713485566, 01:12:46 DEBUG "0" 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG } 01:12:46 INFO [http_cmd]: http status code OK 01:12:46 DEBUG --- status code --- 01:12:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:46 DEBUG --- http response --- 01:12:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:46 DEBUG ----- output ----- 01:12:46 DEBUG { 01:12:46 DEBUG "status": "success", 01:12:46 DEBUG "data": { 01:12:46 DEBUG "resultType": "vector", 01:12:46 DEBUG "result": [ 01:12:46 DEBUG { 01:12:46 DEBUG "metric": { 01:12:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG [...] 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 1713485566, 01:12:46 DEBUG "1906.9333333333334" 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG } 01:12:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485566" 01:12:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485566" 01:12:46 INFO [http_cmd]: http status code OK 01:12:46 DEBUG --- status code --- 01:12:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:46 DEBUG --- http response --- 01:12:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:46 DEBUG ----- output ----- 01:12:46 DEBUG { 01:12:46 DEBUG "status": "success", 01:12:46 DEBUG "data": { 01:12:46 DEBUG "resultType": "vector", 01:12:46 DEBUG "result": [ 01:12:46 DEBUG { 01:12:46 DEBUG "metric": { 01:12:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG [...] 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 1713485566, 01:12:46 DEBUG "6.9694055556654275" 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG } 01:12:46 INFO [http_cmd]: http status code OK 01:12:46 DEBUG --- status code --- 01:12:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:46 DEBUG --- http response --- 01:12:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:46 DEBUG ----- output ----- 01:12:46 DEBUG { 01:12:46 DEBUG "status": "success", 01:12:46 DEBUG "data": { 01:12:46 DEBUG "resultType": "vector", 01:12:46 DEBUG "result": [ 01:12:46 DEBUG { 01:12:46 DEBUG "metric": { 01:12:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG [...] 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 1713485566, 01:12:46 DEBUG "0" 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG } 01:12:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485566" 01:12:46 INFO [http_cmd]: http status code OK 01:12:46 DEBUG --- status code --- 01:12:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:46 DEBUG --- http response --- 01:12:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:12:46 DEBUG ----- output ----- 01:12:46 DEBUG { 01:12:46 DEBUG "status": "success", 01:12:46 DEBUG "data": { 01:12:46 DEBUG "resultType": "vector", 01:12:46 DEBUG "result": [ 01:12:46 DEBUG { 01:12:46 DEBUG "metric": { 01:12:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG [...] 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG "pod": "ds-cts-2" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 1713485566, 01:12:46 DEBUG "0" 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG } 01:12:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485566" 01:12:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485566" 01:12:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485566" 01:12:46 INFO [http_cmd]: http status code OK 01:12:46 DEBUG --- status code --- 01:12:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:46 DEBUG --- http response --- 01:12:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:46 DEBUG ----- output ----- 01:12:46 DEBUG { 01:12:46 DEBUG "status": "success", 01:12:46 DEBUG "data": { 01:12:46 DEBUG "resultType": "vector", 01:12:46 DEBUG "result": [ 01:12:46 DEBUG { 01:12:46 DEBUG "metric": { 01:12:46 DEBUG "pod": "ds-idrepo-0" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG [...] 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG "pod": "ds-idrepo-2" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 1713485566, 01:12:46 DEBUG "0" 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG } 01:12:46 INFO [http_cmd]: http status code OK 01:12:46 DEBUG --- status code --- 01:12:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:46 DEBUG --- http response --- 01:12:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:46 DEBUG ----- output ----- 01:12:46 DEBUG { 01:12:46 DEBUG "status": "success", 01:12:46 DEBUG "data": { 01:12:46 DEBUG "resultType": "vector", 01:12:46 DEBUG "result": [ 01:12:46 DEBUG { 01:12:46 DEBUG "metric": { 01:12:46 DEBUG "pod": "ds-idrepo-0" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG [...] 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG "pod": "ds-idrepo-2" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 1713485566, 01:12:46 DEBUG "0" 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG } 01:12:46 INFO [http_cmd]: http status code OK 01:12:46 DEBUG --- status code --- 01:12:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:46 DEBUG --- http response --- 01:12:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:46 DEBUG ----- output ----- 01:12:46 DEBUG { 01:12:46 DEBUG "status": "success", 01:12:46 DEBUG "data": { 01:12:46 DEBUG "resultType": "vector", 01:12:46 DEBUG "result": [ 01:12:46 DEBUG { 01:12:46 DEBUG "metric": { 01:12:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG [...] 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 1713485566, 01:12:46 DEBUG "0" 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG } 01:12:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485566" 01:12:46 INFO [http_cmd]: http status code OK 01:12:46 DEBUG --- status code --- 01:12:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:46 DEBUG --- http response --- 01:12:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:46 DEBUG ----- output ----- 01:12:46 DEBUG { 01:12:46 DEBUG "status": "success", 01:12:46 DEBUG "data": { 01:12:46 DEBUG "resultType": "vector", 01:12:46 DEBUG "result": [ 01:12:46 DEBUG { 01:12:46 DEBUG "metric": { 01:12:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG [...] 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG 01:12:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:12:46 DEBUG }, 01:12:46 DEBUG "value": [ 01:12:46 DEBUG 1713485566, 01:12:46 DEBUG "1907.3999999999999" 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG ] 01:12:46 DEBUG } 01:12:46 DEBUG } 01:12:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485567" 01:12:47 INFO [http_cmd]: http status code OK 01:12:47 DEBUG --- status code --- 01:12:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:47 DEBUG --- http response --- 01:12:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:47 DEBUG ----- output ----- 01:12:47 DEBUG { 01:12:47 DEBUG "status": "success", 01:12:47 DEBUG "data": { 01:12:47 DEBUG "resultType": "vector", 01:12:47 DEBUG "result": [ 01:12:47 DEBUG { 01:12:47 DEBUG "metric": { 01:12:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG [...] 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 1713485567, 01:12:47 DEBUG "3334463.5" 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG } 01:12:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485567" 01:12:47 INFO [http_cmd]: http status code OK 01:12:47 DEBUG --- status code --- 01:12:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:47 DEBUG --- http response --- 01:12:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:47 DEBUG ----- output ----- 01:12:47 DEBUG { 01:12:47 DEBUG "status": "success", 01:12:47 DEBUG "data": { 01:12:47 DEBUG "resultType": "vector", 01:12:47 DEBUG "result": [ 01:12:47 DEBUG { 01:12:47 DEBUG "metric": { 01:12:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG [...] 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 1713485567, 01:12:47 DEBUG "1907" 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG } 01:12:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485567" 01:12:47 INFO [http_cmd]: http status code OK 01:12:47 DEBUG --- status code --- 01:12:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:47 DEBUG --- http response --- 01:12:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:47 DEBUG ----- output ----- 01:12:47 DEBUG { 01:12:47 DEBUG "status": "success", 01:12:47 DEBUG "data": { 01:12:47 DEBUG "resultType": "vector", 01:12:47 DEBUG "result": [ 01:12:47 DEBUG { 01:12:47 DEBUG "metric": { 01:12:47 DEBUG "instance": "10.142.0.107:9100" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG [...] 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG "instance": "10.142.0.43:9100" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 1713485567, 01:12:47 DEBUG "47231114.63333333" 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG } 01:12:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485567" 01:12:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485567" 01:12:47 INFO [http_cmd]: http status code OK 01:12:47 DEBUG --- status code --- 01:12:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:47 DEBUG --- http response --- 01:12:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485567" 01:12:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:12:47 DEBUG ----- output ----- 01:12:47 DEBUG { 01:12:47 DEBUG "status": "success", 01:12:47 DEBUG "data": { 01:12:47 DEBUG "resultType": "vector", 01:12:47 DEBUG "result": [ 01:12:47 DEBUG { 01:12:47 DEBUG "metric": { 01:12:47 DEBUG "pod": "ds-cts-0" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG [...] 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG "pod": "ds-cts-2" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 1713485567, 01:12:47 DEBUG "0.000033333333333333335" 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG } 01:12:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485567" 01:12:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485567" 01:12:47 INFO [http_cmd]: http status code OK 01:12:47 DEBUG --- status code --- 01:12:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:47 DEBUG --- http response --- 01:12:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:12:47 DEBUG ----- output ----- 01:12:47 DEBUG { 01:12:47 DEBUG "status": "success", 01:12:47 DEBUG "data": { 01:12:47 DEBUG "resultType": "vector", 01:12:47 DEBUG "result": [ 01:12:47 DEBUG { 01:12:47 DEBUG "metric": { 01:12:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG [...] 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 1713485567, 01:12:47 DEBUG "20.076" 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG } 01:12:47 INFO [http_cmd]: http status code OK 01:12:47 DEBUG --- status code --- 01:12:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:47 DEBUG --- http response --- 01:12:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:12:47 DEBUG ----- output ----- 01:12:47 DEBUG { 01:12:47 DEBUG "status": "success", 01:12:47 DEBUG "data": { 01:12:47 DEBUG "resultType": "vector", 01:12:47 DEBUG "result": [ 01:12:47 DEBUG { 01:12:47 DEBUG "metric": { 01:12:47 DEBUG "pod": "ds-cts-0" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG [...] 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG "pod": "ds-cts-2" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 1713485567, 01:12:47 DEBUG "793.2" 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG } 01:12:47 INFO [http_cmd]: http status code OK 01:12:47 DEBUG --- status code --- 01:12:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:47 DEBUG --- http response --- 01:12:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:47 DEBUG ----- output ----- 01:12:47 DEBUG { 01:12:47 DEBUG "status": "success", 01:12:47 DEBUG "data": { 01:12:47 DEBUG "resultType": "vector", 01:12:47 DEBUG "result": [ 01:12:47 DEBUG { 01:12:47 DEBUG "metric": { 01:12:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG [...] 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 1713485567, 01:12:47 DEBUG "1082916864" 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG } 01:12:47 INFO [http_cmd]: http status code OK 01:12:47 DEBUG --- status code --- 01:12:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:12:47 DEBUG --- http response --- 01:12:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:12:47 DEBUG ----- output ----- 01:12:47 DEBUG { 01:12:47 DEBUG "status": "success", 01:12:47 DEBUG "data": { 01:12:47 DEBUG "resultType": "vector", 01:12:47 DEBUG "result": [ 01:12:47 DEBUG { 01:12:47 DEBUG "metric": { 01:12:47 DEBUG "instance": "10.142.0.107:9100" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG [...] 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG 01:12:47 DEBUG "instance": "10.142.0.43:9100" 01:12:47 DEBUG }, 01:12:47 DEBUG "value": [ 01:12:47 DEBUG 1713485567, 01:12:47 DEBUG "1.0043333333333322" 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG ] 01:12:47 DEBUG } 01:12:47 DEBUG } 01:12:47 INFO 01:12:47 INFO [loop_until]: kubectl --namespace=xlou top pods 01:12:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:12:47 INFO [loop_until]: OK (rc = 0) 01:12:47 DEBUG --- stdout --- 01:12:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 11196m 9407Mi am-6fd554f8f7-hvvkw 8469m 8886Mi am-6fd554f8f7-pmj4b 9197m 9067Mi ds-cts-0 9308m 12420Mi ds-cts-1 8687m 12377Mi ds-cts-2 8496m 12367Mi ds-idrepo-0 4784m 3243Mi ds-idrepo-1 4728m 3125Mi ds-idrepo-2 4572m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6861m 937Mi 01:12:47 DEBUG --- stderr --- 01:12:47 DEBUG 01:12:47 INFO 01:12:47 INFO [loop_until]: kubectl --namespace=xlou top node 01:12:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:12:47 INFO [loop_until]: OK (rc = 0) 01:12:47 DEBUG --- stdout --- 01:12:47 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5917Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4642m 15% 6637Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8437m 28% 11613Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3230Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 9253m 30% 14894Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5235Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10921m 36% 12194Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28078m 94% 3131Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10002m 33% 11294Mi 10% gke-xlou-cdm-ds-0f601751-j488 5045m 16% 6085Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8333m 27% 16395Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5294m 17% 6765Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8402m 28% 14317Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7120m 44% 4668Mi 7% 01:12:47 DEBUG --- stderr --- 01:12:47 DEBUG 01:13:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485625" 01:13:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485625" 01:13:45 ERROR [http_cmd]: ERROR 01:13:45 ERROR --- status code --- 01:13:45 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:13:45 ERROR --- http response --- 01:13:45 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:13:45 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:13:45 INFO [http_cmd]: http status code OK 01:13:45 DEBUG --- status code --- 01:13:45 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:45 DEBUG --- http response --- 01:13:45 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:45 DEBUG ----- output ----- 01:13:45 DEBUG { 01:13:45 DEBUG "status": "success", 01:13:45 DEBUG "data": { 01:13:45 DEBUG "resultType": "vector", 01:13:45 DEBUG "result": [ 01:13:45 DEBUG { 01:13:45 DEBUG "metric": { 01:13:45 DEBUG "instance": "10.142.0.107:9100" 01:13:45 DEBUG }, 01:13:45 DEBUG "value": [ 01:13:45 DEBUG 01:13:45 DEBUG 01:13:45 DEBUG 01:13:45 DEBUG [...] 01:13:45 DEBUG 01:13:45 DEBUG 01:13:45 DEBUG 01:13:45 DEBUG "instance": "10.142.0.43:9100" 01:13:45 DEBUG }, 01:13:45 DEBUG "value": [ 01:13:45 DEBUG 1713485625, 01:13:45 DEBUG "0.9572333333333356" 01:13:45 DEBUG ] 01:13:45 DEBUG } 01:13:45 DEBUG ] 01:13:45 DEBUG } 01:13:45 DEBUG } 01:13:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485625" 01:13:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485625" 01:13:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485625" 01:13:45 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485625" 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG --- http response --- 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG { 01:13:46 DEBUG { 01:13:46 DEBUG { 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "data": { 01:13:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:13:46 DEBUG "data": { 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "data": { 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "result": [ 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "result": [ 01:13:46 DEBUG { 01:13:46 DEBUG "result": [ 01:13:46 DEBUG { 01:13:46 DEBUG "metric": { 01:13:46 DEBUG { 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:13:46 DEBUG }, 01:13:46 DEBUG "instance": "10.142.0.107:9100" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG [...] 01:13:46 DEBUG 01:13:46 DEBUG [...] 01:13:46 DEBUG 01:13:46 DEBUG [...] 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:13:46 DEBUG 01:13:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:13:46 DEBUG }, 01:13:46 DEBUG "instance": "10.142.0.43:9100" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 1713485625, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 1713485625, 01:13:46 DEBUG "934.4664347230937" 01:13:46 DEBUG 1713485625, 01:13:46 DEBUG "670.7374344444445" 01:13:46 DEBUG ] 01:13:46 DEBUG "343594509312" 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485626" 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG { 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "data": { 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "result": [ 01:13:46 DEBUG { 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "pod": "ds-idrepo-0" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG [...] 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG "pod": "ds-cts-2" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 1713485626, 01:13:46 DEBUG "0" 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485626" 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG { 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "data": { 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "result": [ 01:13:46 DEBUG { 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "instance": "10.142.0.107:9100" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG [...] 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG "instance": "10.142.0.43:9100" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 1713485626, 01:13:46 DEBUG "50083499.099999994" 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485626" 01:13:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485626" 01:13:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485626" 01:13:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485626" 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG { 01:13:46 DEBUG { 01:13:46 DEBUG { 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "data": { 01:13:46 DEBUG "data": { 01:13:46 DEBUG "data": { 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "result": [ 01:13:46 DEBUG "result": [ 01:13:46 DEBUG "result": [ 01:13:46 DEBUG { 01:13:46 DEBUG { 01:13:46 DEBUG { 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "pod": "ds-idrepo-0" 01:13:46 DEBUG "pod": "ds-cts-0" 01:13:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:13:46 DEBUG }, 01:13:46 DEBUG }, 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG "value": [ 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG 01:13:46 DEBUG [...] 01:13:46 DEBUG [...] 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG [...] 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG 01:13:46 DEBUG "pod": "ds-idrepo-2" 01:13:46 DEBUG "pod": "ds-cts-2" 01:13:46 DEBUG { 01:13:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:13:46 DEBUG }, 01:13:46 DEBUG }, 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG "value": [ 01:13:46 DEBUG "data": { 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 1713485626, 01:13:46 DEBUG 1713485626, 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG 1713485626, 01:13:46 DEBUG "0" 01:13:46 DEBUG "0" 01:13:46 DEBUG "result": [ 01:13:46 DEBUG { 01:13:46 DEBUG ] 01:13:46 DEBUG ] 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "0" 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:13:46 DEBUG ] 01:13:46 DEBUG ] 01:13:46 DEBUG ] 01:13:46 DEBUG }, 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 DEBUG "value": [ 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 DEBUG 01:13:46 DEBUG } 01:13:46 DEBUG 01:13:46 DEBUG } 01:13:46 DEBUG 01:13:46 DEBUG [...] 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 1713485626, 01:13:46 DEBUG "1813.7" 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485626" 01:13:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485626" 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG { 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "data": { 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "result": [ 01:13:46 DEBUG { 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG [...] 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 1713485626, 01:13:46 DEBUG "6.9408990297132" 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG { 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "data": { 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "result": [ 01:13:46 DEBUG { 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG [...] 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 1713485626, 01:13:46 DEBUG "0" 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485626" 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG { 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "data": { 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "result": [ 01:13:46 DEBUG { 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG [...] 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 1713485626, 01:13:46 DEBUG "0" 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485626" 01:13:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485626" 01:13:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485626" 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG { 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "data": { 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "result": [ 01:13:46 DEBUG { 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "pod": "ds-idrepo-0" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG [...] 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG "pod": "ds-idrepo-2" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 1713485626, 01:13:46 DEBUG "0" 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG { 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "data": { 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "result": [ 01:13:46 DEBUG { 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "pod": "ds-idrepo-0" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG [...] 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG "pod": "ds-idrepo-2" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 1713485626, 01:13:46 DEBUG "0" 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG { 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "data": { 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "result": [ 01:13:46 DEBUG { 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG [...] 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 1713485626, 01:13:46 DEBUG "0" 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485626" 01:13:46 INFO [http_cmd]: http status code OK 01:13:46 DEBUG --- status code --- 01:13:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:46 DEBUG --- http response --- 01:13:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:46 DEBUG ----- output ----- 01:13:46 DEBUG { 01:13:46 DEBUG "status": "success", 01:13:46 DEBUG "data": { 01:13:46 DEBUG "resultType": "vector", 01:13:46 DEBUG "result": [ 01:13:46 DEBUG { 01:13:46 DEBUG "metric": { 01:13:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG [...] 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG 01:13:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:13:46 DEBUG }, 01:13:46 DEBUG "value": [ 01:13:46 DEBUG 1713485626, 01:13:46 DEBUG "1814.8999999999999" 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG ] 01:13:46 DEBUG } 01:13:46 DEBUG } 01:13:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485627" 01:13:47 INFO [http_cmd]: http status code OK 01:13:47 DEBUG --- status code --- 01:13:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:47 DEBUG --- http response --- 01:13:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:47 DEBUG ----- output ----- 01:13:47 DEBUG { 01:13:47 DEBUG "status": "success", 01:13:47 DEBUG "data": { 01:13:47 DEBUG "resultType": "vector", 01:13:47 DEBUG "result": [ 01:13:47 DEBUG { 01:13:47 DEBUG "metric": { 01:13:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG [...] 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 1713485627, 01:13:47 DEBUG "3431644.5" 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG } 01:13:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485627" 01:13:47 INFO [http_cmd]: http status code OK 01:13:47 DEBUG --- status code --- 01:13:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:47 DEBUG --- http response --- 01:13:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:47 DEBUG ----- output ----- 01:13:47 DEBUG { 01:13:47 DEBUG "status": "success", 01:13:47 DEBUG "data": { 01:13:47 DEBUG "resultType": "vector", 01:13:47 DEBUG "result": [ 01:13:47 DEBUG { 01:13:47 DEBUG "metric": { 01:13:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG [...] 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 1713485627, 01:13:47 DEBUG "1813.8333333333333" 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG } 01:13:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485627" 01:13:47 INFO [http_cmd]: http status code OK 01:13:47 DEBUG --- status code --- 01:13:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:47 DEBUG --- http response --- 01:13:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:47 DEBUG ----- output ----- 01:13:47 DEBUG { 01:13:47 DEBUG "status": "success", 01:13:47 DEBUG "data": { 01:13:47 DEBUG "resultType": "vector", 01:13:47 DEBUG "result": [ 01:13:47 DEBUG { 01:13:47 DEBUG "metric": { 01:13:47 DEBUG "instance": "10.142.0.107:9100" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG [...] 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG "instance": "10.142.0.43:9100" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 1713485627, 01:13:47 DEBUG "50089521.8" 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG } 01:13:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485627" 01:13:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485627" 01:13:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485627" 01:13:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485627" 01:13:47 INFO [http_cmd]: http status code OK 01:13:47 INFO [http_cmd]: http status code OK 01:13:47 DEBUG --- status code --- 01:13:47 DEBUG --- status code --- 01:13:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:47 DEBUG --- http response --- 01:13:47 DEBUG --- http response --- 01:13:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:13:47 DEBUG ----- output ----- 01:13:47 DEBUG { 01:13:47 DEBUG "status": "success", 01:13:47 DEBUG "data": { 01:13:47 DEBUG "resultType": "vector", 01:13:47 DEBUG "result": [ 01:13:47 DEBUG { 01:13:47 DEBUG "metric": { 01:13:47 DEBUG "pod": "ds-cts-0" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG [...] 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG "pod": "ds-cts-2" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 1713485627, 01:13:47 DEBUG "0.00031548888888888876" 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG } 01:13:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:47 DEBUG ----- output ----- 01:13:47 DEBUG { 01:13:47 DEBUG "status": "success", 01:13:47 DEBUG "data": { 01:13:47 DEBUG "resultType": "vector", 01:13:47 DEBUG "result": [ 01:13:47 DEBUG { 01:13:47 DEBUG "metric": { 01:13:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG [...] 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 1713485627, 01:13:47 DEBUG "1085276160" 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG } 01:13:47 INFO [http_cmd]: http status code OK 01:13:47 DEBUG --- status code --- 01:13:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:47 DEBUG --- http response --- 01:13:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:13:47 DEBUG ----- output ----- 01:13:47 DEBUG { 01:13:47 DEBUG "status": "success", 01:13:47 DEBUG "data": { 01:13:47 DEBUG "resultType": "vector", 01:13:47 DEBUG "result": [ 01:13:47 DEBUG { 01:13:47 DEBUG "metric": { 01:13:47 DEBUG "instance": "10.142.0.107:9100" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG [...] 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG "instance": "10.142.0.43:9100" 01:13:47 DEBUG }, 01:13:47 INFO [http_cmd]: http status code OK 01:13:47 DEBUG "value": [ 01:13:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485627" 01:13:47 DEBUG --- status code --- 01:13:47 DEBUG 1713485627, 01:13:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:47 DEBUG "1.752" 01:13:47 DEBUG --- http response --- 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:13:47 DEBUG ] 01:13:47 DEBUG ----- output ----- 01:13:47 DEBUG } 01:13:47 DEBUG { 01:13:47 DEBUG } 01:13:47 DEBUG "status": "success", 01:13:47 DEBUG "data": { 01:13:47 DEBUG "resultType": "vector", 01:13:47 DEBUG "result": [ 01:13:47 DEBUG { 01:13:47 DEBUG "metric": { 01:13:47 DEBUG "pod": "ds-cts-0" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG [...] 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG "pod": "ds-cts-2" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 1713485627, 01:13:47 DEBUG "4595.533333333333" 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG } 01:13:47 INFO [http_cmd]: http status code OK 01:13:47 DEBUG --- status code --- 01:13:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:13:47 DEBUG --- http response --- 01:13:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:13:47 DEBUG ----- output ----- 01:13:47 DEBUG { 01:13:47 DEBUG "status": "success", 01:13:47 DEBUG "data": { 01:13:47 DEBUG "resultType": "vector", 01:13:47 DEBUG "result": [ 01:13:47 DEBUG { 01:13:47 DEBUG "metric": { 01:13:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG [...] 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG 01:13:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:13:47 DEBUG }, 01:13:47 DEBUG "value": [ 01:13:47 DEBUG 1713485627, 01:13:47 DEBUG "20.076" 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG ] 01:13:47 DEBUG } 01:13:47 DEBUG } 01:13:47 INFO 01:13:47 INFO [loop_until]: kubectl --namespace=xlou top pods 01:13:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:13:47 INFO [loop_until]: OK (rc = 0) 01:13:47 DEBUG --- stdout --- 01:13:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 12196m 9426Mi am-6fd554f8f7-hvvkw 8700m 8887Mi am-6fd554f8f7-pmj4b 9644m 9070Mi ds-cts-0 10892m 12409Mi ds-cts-1 9274m 12410Mi ds-cts-2 11022m 12465Mi ds-idrepo-0 4901m 3246Mi ds-idrepo-1 4949m 3126Mi ds-idrepo-2 4708m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 8m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6754m 940Mi 01:13:47 DEBUG --- stderr --- 01:13:47 DEBUG 01:13:47 INFO 01:13:47 INFO [loop_until]: kubectl --namespace=xlou top node 01:13:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:13:48 INFO [loop_until]: OK (rc = 0) 01:13:48 DEBUG --- stdout --- 01:13:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5910Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4544m 15% 6652Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9143m 30% 11612Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3239Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 11126m 37% 14865Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5216Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 11797m 39% 12209Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 28182m 94% 3145Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 10591m 35% 11291Mi 10% gke-xlou-cdm-ds-0f601751-j488 5166m 17% 6107Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 10282m 34% 16429Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 5350m 17% 6754Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 10602m 35% 14390Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7102m 44% 4669Mi 7% 01:13:48 DEBUG --- stderr --- 01:13:48 DEBUG 01:13:56 WARNING File /tmp/lodemon_data-20240419_004139/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [19/Apr/2024 01:13:56] "GET /monitoring/average?start_time=20240418_234445&stop_time=20240419_001155 HTTP/1.1" 200 - 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485686" 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485686" 01:14:46 ERROR [http_cmd]: ERROR 01:14:46 INFO [http_cmd]: http status code OK 01:14:46 DEBUG --- status code --- 01:14:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:46 DEBUG --- http response --- 01:14:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:46 DEBUG ----- output ----- 01:14:46 DEBUG { 01:14:46 DEBUG "status": "success", 01:14:46 DEBUG "data": { 01:14:46 DEBUG "resultType": "vector", 01:14:46 DEBUG "result": [ 01:14:46 DEBUG { 01:14:46 DEBUG "metric": { 01:14:46 DEBUG "instance": "10.142.0.107:9100" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG [...] 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG "instance": "10.142.0.43:9100" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 1713485686, 01:14:46 DEBUG "0.8551666666666686" 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG } 01:14:46 ERROR --- status code --- 01:14:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:14:46 ERROR --- http response --- 01:14:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:14:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485686" 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485686" 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485686" 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485686" 01:14:46 INFO [http_cmd]: http status code OK 01:14:46 DEBUG --- status code --- 01:14:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:46 DEBUG --- http response --- 01:14:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:14:46 INFO [http_cmd]: http status code OK 01:14:46 DEBUG --- status code --- 01:14:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:46 DEBUG --- http response --- 01:14:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 01:14:46 DEBUG ----- output ----- 01:14:46 DEBUG { 01:14:46 DEBUG "status": "success", 01:14:46 DEBUG "data": { 01:14:46 DEBUG "resultType": "vector", 01:14:46 DEBUG "result": [ 01:14:46 DEBUG { 01:14:46 DEBUG "metric": { 01:14:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG [...] 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG "pod": "ds-cts-0" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 1713485686, 01:14:46 DEBUG "3427128.7660375223" 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG } 01:14:46 INFO [http_cmd]: http status code OK 01:14:46 DEBUG --- status code --- 01:14:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:46 DEBUG --- http response --- 01:14:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:46 DEBUG ----- output ----- 01:14:46 DEBUG { 01:14:46 DEBUG "status": "success", 01:14:46 DEBUG "data": { 01:14:46 DEBUG "resultType": "vector", 01:14:46 DEBUG "result": [ 01:14:46 DEBUG { 01:14:46 DEBUG "metric": { 01:14:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG [...] 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 1713485686, 01:14:46 DEBUG "1914.3" 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG } 01:14:46 INFO [http_cmd]: http status code OK 01:14:46 DEBUG --- status code --- 01:14:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:46 DEBUG --- http response --- 01:14:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:46 DEBUG ----- output ----- 01:14:46 DEBUG { 01:14:46 DEBUG "status": "success", 01:14:46 DEBUG "data": { 01:14:46 DEBUG "resultType": "vector", 01:14:46 DEBUG "result": [ 01:14:46 DEBUG { 01:14:46 DEBUG "metric": { 01:14:46 DEBUG "instance": "10.142.0.107:9100" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG [...] 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG "instance": "10.142.0.43:9100" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 1713485686, 01:14:46 DEBUG "344706112512" 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG } 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485686" 01:14:46 INFO [http_cmd]: http status code OK 01:14:46 DEBUG --- status code --- 01:14:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:46 DEBUG --- http response --- 01:14:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:14:46 DEBUG ----- output ----- 01:14:46 DEBUG { 01:14:46 DEBUG "status": "success", 01:14:46 DEBUG "data": { 01:14:46 DEBUG "resultType": "vector", 01:14:46 DEBUG "result": [ 01:14:46 DEBUG { 01:14:46 DEBUG "metric": { 01:14:46 DEBUG "pod": "ds-idrepo-0" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG [...] 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG "pod": "ds-cts-2" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 1713485686, 01:14:46 DEBUG "0" 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG } 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485686" 01:14:46 INFO [http_cmd]: http status code OK 01:14:46 DEBUG --- status code --- 01:14:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:46 DEBUG --- http response --- 01:14:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:46 DEBUG ----- output ----- 01:14:46 DEBUG { 01:14:46 DEBUG "status": "success", 01:14:46 DEBUG "data": { 01:14:46 DEBUG "resultType": "vector", 01:14:46 DEBUG "result": [ 01:14:46 DEBUG { 01:14:46 DEBUG "metric": { 01:14:46 DEBUG "instance": "10.142.0.107:9100" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG [...] 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG "instance": "10.142.0.43:9100" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 1713485686, 01:14:46 DEBUG "27336790.433333334" 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG } 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485686" 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485686" 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485686" 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485686" 01:14:46 INFO [http_cmd]: http status code OK 01:14:46 DEBUG --- status code --- 01:14:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:46 DEBUG --- http response --- 01:14:46 INFO [http_cmd]: http status code OK 01:14:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:14:46 DEBUG --- status code --- 01:14:46 DEBUG ----- output ----- 01:14:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:46 DEBUG { 01:14:46 DEBUG --- http response --- 01:14:46 DEBUG "status": "success", 01:14:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:46 DEBUG "data": { 01:14:46 DEBUG ----- output ----- 01:14:46 DEBUG "resultType": "vector", 01:14:46 DEBUG { 01:14:46 DEBUG "result": [ 01:14:46 DEBUG "status": "success", 01:14:46 DEBUG { 01:14:46 DEBUG "data": { 01:14:46 DEBUG "metric": { 01:14:46 DEBUG "resultType": "vector", 01:14:46 DEBUG "pod": "ds-cts-0" 01:14:46 DEBUG "result": [ 01:14:46 DEBUG }, 01:14:46 DEBUG { 01:14:46 DEBUG "value": [ 01:14:46 DEBUG "metric": { 01:14:46 DEBUG 01:14:46 DEBUG "pod": "ds-idrepo-0" 01:14:46 DEBUG 01:14:46 DEBUG }, 01:14:46 DEBUG 01:14:46 DEBUG "value": [ 01:14:46 DEBUG [...] 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG [...] 01:14:46 DEBUG "pod": "ds-cts-2" 01:14:46 DEBUG 01:14:46 DEBUG }, 01:14:46 DEBUG 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 01:14:46 DEBUG 1713485686, 01:14:46 DEBUG "pod": "ds-idrepo-2" 01:14:46 DEBUG "0" 01:14:46 DEBUG }, 01:14:46 DEBUG ] 01:14:46 DEBUG "value": [ 01:14:46 DEBUG } 01:14:46 DEBUG 1713485686, 01:14:46 DEBUG "0" 01:14:46 DEBUG ] 01:14:46 DEBUG ] 01:14:46 INFO [http_cmd]: http status code OK 01:14:46 INFO [http_cmd]: http status code OK 01:14:46 DEBUG } 01:14:46 DEBUG } 01:14:46 DEBUG --- status code --- 01:14:46 DEBUG --- status code --- 01:14:46 DEBUG } 01:14:46 DEBUG ] 01:14:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:46 DEBUG } 01:14:46 DEBUG --- http response --- 01:14:46 DEBUG --- http response --- 01:14:46 DEBUG } 01:14:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:46 DEBUG ----- output ----- 01:14:46 DEBUG ----- output ----- 01:14:46 DEBUG { 01:14:46 DEBUG { 01:14:46 DEBUG "status": "success", 01:14:46 DEBUG "status": "success", 01:14:46 DEBUG "data": { 01:14:46 DEBUG "resultType": "vector", 01:14:46 DEBUG "data": { 01:14:46 DEBUG "result": [ 01:14:46 DEBUG "resultType": "vector", 01:14:46 DEBUG { 01:14:46 DEBUG "result": [ 01:14:46 DEBUG "metric": { 01:14:46 DEBUG { 01:14:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:14:46 DEBUG "metric": { 01:14:46 DEBUG }, 01:14:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:14:46 DEBUG "value": [ 01:14:46 DEBUG }, 01:14:46 DEBUG 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG [...] 01:14:46 DEBUG [...] 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:14:46 DEBUG "pod": "ds-cts-0" 01:14:46 DEBUG }, 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 1713485686, 01:14:46 DEBUG 1713485686, 01:14:46 DEBUG "494.8666666666667" 01:14:46 DEBUG "1501673.2367057758" 01:14:46 DEBUG ] 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG } 01:14:46 DEBUG ] 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG } 01:14:46 DEBUG } 01:14:46 DEBUG } 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485686" 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485686" 01:14:46 INFO [http_cmd]: http status code OK 01:14:46 DEBUG --- status code --- 01:14:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:46 DEBUG --- http response --- 01:14:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:46 DEBUG ----- output ----- 01:14:46 DEBUG { 01:14:46 DEBUG "status": "success", 01:14:46 DEBUG "data": { 01:14:46 DEBUG "resultType": "vector", 01:14:46 DEBUG "result": [ 01:14:46 DEBUG { 01:14:46 DEBUG "metric": { 01:14:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG [...] 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 1713485686, 01:14:46 DEBUG "3.28196480643688" 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG } 01:14:46 INFO [http_cmd]: http status code OK 01:14:46 DEBUG --- status code --- 01:14:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:46 DEBUG --- http response --- 01:14:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:46 DEBUG ----- output ----- 01:14:46 DEBUG { 01:14:46 DEBUG "status": "success", 01:14:46 DEBUG "data": { 01:14:46 DEBUG "resultType": "vector", 01:14:46 DEBUG "result": [ 01:14:46 DEBUG { 01:14:46 DEBUG "metric": { 01:14:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG [...] 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 1713485686, 01:14:46 DEBUG "0" 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG } 01:14:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485686" 01:14:46 INFO [http_cmd]: http status code OK 01:14:46 DEBUG --- status code --- 01:14:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:46 DEBUG --- http response --- 01:14:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:46 DEBUG ----- output ----- 01:14:46 DEBUG { 01:14:46 DEBUG "status": "success", 01:14:46 DEBUG "data": { 01:14:46 DEBUG "resultType": "vector", 01:14:46 DEBUG "result": [ 01:14:46 DEBUG { 01:14:46 DEBUG "metric": { 01:14:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG [...] 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG 01:14:46 DEBUG "pod": "ds-cts-0" 01:14:46 DEBUG }, 01:14:46 DEBUG "value": [ 01:14:46 DEBUG 1713485686, 01:14:46 DEBUG "5.2246120690392575" 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG ] 01:14:46 DEBUG } 01:14:46 DEBUG } 01:14:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485687" 01:14:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485687" 01:14:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485687" 01:14:47 INFO [http_cmd]: http status code OK 01:14:47 INFO [http_cmd]: http status code OK 01:14:47 DEBUG --- status code --- 01:14:47 DEBUG --- status code --- 01:14:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:47 DEBUG --- http response --- 01:14:47 DEBUG --- http response --- 01:14:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:47 DEBUG ----- output ----- 01:14:47 DEBUG ----- output ----- 01:14:47 DEBUG { 01:14:47 DEBUG { 01:14:47 DEBUG "status": "success", 01:14:47 DEBUG "status": "success", 01:14:47 DEBUG "data": { 01:14:47 DEBUG "data": { 01:14:47 DEBUG "resultType": "vector", 01:14:47 DEBUG "resultType": "vector", 01:14:47 DEBUG "result": [ 01:14:47 DEBUG "result": [ 01:14:47 DEBUG { 01:14:47 DEBUG { 01:14:47 DEBUG "metric": { 01:14:47 DEBUG "metric": { 01:14:47 DEBUG "pod": "ds-idrepo-0" 01:14:47 DEBUG "pod": "ds-idrepo-0" 01:14:47 DEBUG }, 01:14:47 DEBUG }, 01:14:47 INFO [http_cmd]: http status code OK 01:14:47 DEBUG "value": [ 01:14:47 DEBUG "value": [ 01:14:47 DEBUG --- status code --- 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG --- http response --- 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 01:14:47 DEBUG [...] 01:14:47 DEBUG [...] 01:14:47 DEBUG ----- output ----- 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG { 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG "status": "success", 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG "data": { 01:14:47 DEBUG "pod": "ds-idrepo-2" 01:14:47 DEBUG "pod": "ds-idrepo-2" 01:14:47 DEBUG "resultType": "vector", 01:14:47 DEBUG }, 01:14:47 DEBUG }, 01:14:47 DEBUG "result": [ 01:14:47 DEBUG "value": [ 01:14:47 DEBUG "value": [ 01:14:47 DEBUG { 01:14:47 DEBUG "metric": { 01:14:47 DEBUG 1713485687, 01:14:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:14:47 DEBUG 1713485687, 01:14:47 DEBUG "0" 01:14:47 DEBUG }, 01:14:47 DEBUG "0" 01:14:47 DEBUG ] 01:14:47 DEBUG "value": [ 01:14:47 DEBUG ] 01:14:47 DEBUG } 01:14:47 DEBUG 01:14:47 DEBUG } 01:14:47 DEBUG ] 01:14:47 DEBUG 01:14:47 DEBUG ] 01:14:47 DEBUG } 01:14:47 DEBUG 01:14:47 DEBUG } 01:14:47 DEBUG } 01:14:47 DEBUG [...] 01:14:47 DEBUG } 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG "pod": "ds-cts-0" 01:14:47 DEBUG }, 01:14:47 DEBUG "value": [ 01:14:47 DEBUG 1713485687, 01:14:47 DEBUG "19932080.738619663" 01:14:47 DEBUG ] 01:14:47 DEBUG } 01:14:47 DEBUG ] 01:14:47 DEBUG } 01:14:47 DEBUG } 01:14:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485687" 01:14:47 INFO [http_cmd]: http status code OK 01:14:47 DEBUG --- status code --- 01:14:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:47 DEBUG --- http response --- 01:14:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:47 DEBUG ----- output ----- 01:14:47 DEBUG { 01:14:47 DEBUG "status": "success", 01:14:47 DEBUG "data": { 01:14:47 DEBUG "resultType": "vector", 01:14:47 DEBUG "result": [ 01:14:47 DEBUG { 01:14:47 DEBUG "metric": { 01:14:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:14:47 DEBUG }, 01:14:47 DEBUG "value": [ 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG [...] 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:14:47 DEBUG }, 01:14:47 DEBUG "value": [ 01:14:47 DEBUG 1713485687, 01:14:47 DEBUG "494.23333333333335" 01:14:47 DEBUG ] 01:14:47 DEBUG } 01:14:47 DEBUG ] 01:14:47 DEBUG } 01:14:47 DEBUG } 01:14:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485687" 01:14:47 INFO [http_cmd]: http status code OK 01:14:47 DEBUG --- status code --- 01:14:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:47 DEBUG --- http response --- 01:14:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485687" 01:14:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:47 DEBUG ----- output ----- 01:14:47 DEBUG { 01:14:47 DEBUG "status": "success", 01:14:47 DEBUG "data": { 01:14:47 DEBUG "resultType": "vector", 01:14:47 DEBUG "result": [ 01:14:47 DEBUG { 01:14:47 DEBUG "metric": { 01:14:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:14:47 DEBUG }, 01:14:47 DEBUG "value": [ 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG [...] 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:14:47 DEBUG }, 01:14:47 DEBUG "value": [ 01:14:47 DEBUG 1713485687, 01:14:47 DEBUG "3364496.5" 01:14:47 DEBUG ] 01:14:47 DEBUG } 01:14:47 DEBUG ] 01:14:47 DEBUG } 01:14:47 DEBUG } 01:14:47 INFO [http_cmd]: http status code OK 01:14:47 DEBUG --- status code --- 01:14:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:47 DEBUG --- http response --- 01:14:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:47 DEBUG ----- output ----- 01:14:47 DEBUG { 01:14:47 DEBUG "status": "success", 01:14:47 DEBUG "data": { 01:14:47 DEBUG "resultType": "vector", 01:14:47 DEBUG "result": [ 01:14:47 DEBUG { 01:14:47 DEBUG "metric": { 01:14:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:14:47 DEBUG }, 01:14:47 DEBUG "value": [ 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG [...] 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:14:47 DEBUG }, 01:14:47 DEBUG "value": [ 01:14:47 DEBUG 1713485687, 01:14:47 DEBUG "494.56666666666666" 01:14:47 DEBUG ] 01:14:47 DEBUG } 01:14:47 DEBUG ] 01:14:47 DEBUG } 01:14:47 DEBUG } 01:14:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485687" 01:14:47 INFO [http_cmd]: http status code OK 01:14:47 DEBUG --- status code --- 01:14:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:47 DEBUG --- http response --- 01:14:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:47 DEBUG ----- output ----- 01:14:47 DEBUG { 01:14:47 DEBUG "status": "success", 01:14:47 DEBUG "data": { 01:14:47 DEBUG "resultType": "vector", 01:14:47 DEBUG "result": [ 01:14:47 DEBUG { 01:14:47 DEBUG "metric": { 01:14:47 DEBUG "instance": "10.142.0.107:9100" 01:14:47 DEBUG }, 01:14:47 DEBUG "value": [ 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG [...] 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG "instance": "10.142.0.43:9100" 01:14:47 DEBUG }, 01:14:47 DEBUG "value": [ 01:14:47 DEBUG 1713485687, 01:14:47 DEBUG "27342941.53333333" 01:14:47 DEBUG ] 01:14:47 DEBUG } 01:14:47 DEBUG ] 01:14:47 DEBUG } 01:14:47 DEBUG } 01:14:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485687" 01:14:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485687" 01:14:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485687" 01:14:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485687" 01:14:47 INFO [http_cmd]: http status code OK 01:14:47 DEBUG --- status code --- 01:14:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485687" 01:14:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:47 INFO [http_cmd]: http status code OK 01:14:47 DEBUG --- http response --- 01:14:47 DEBUG --- status code --- 01:14:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:14:47 DEBUG --- http response --- 01:14:47 DEBUG ----- output ----- 01:14:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:47 DEBUG { 01:14:47 INFO [http_cmd]: http status code OK 01:14:47 DEBUG ----- output ----- 01:14:47 DEBUG "status": "success", 01:14:47 DEBUG --- status code --- 01:14:47 DEBUG { 01:14:47 DEBUG "data": { 01:14:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:47 DEBUG "status": "success", 01:14:47 DEBUG "resultType": "vector", 01:14:47 DEBUG --- http response --- 01:14:47 DEBUG "data": { 01:14:47 DEBUG "result": [ 01:14:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:14:47 DEBUG "resultType": "vector", 01:14:47 DEBUG { 01:14:47 DEBUG ----- output ----- 01:14:47 DEBUG "result": [ 01:14:47 DEBUG "metric": { 01:14:47 DEBUG { 01:14:47 DEBUG { 01:14:47 DEBUG "pod": "ds-cts-0" 01:14:47 DEBUG "status": "success", 01:14:47 DEBUG "metric": { 01:14:47 INFO [http_cmd]: http status code OK 01:14:47 DEBUG }, 01:14:47 DEBUG "data": { 01:14:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:14:47 DEBUG --- status code --- 01:14:47 DEBUG "value": [ 01:14:47 DEBUG "resultType": "vector", 01:14:47 DEBUG }, 01:14:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:47 DEBUG 01:14:47 DEBUG "result": [ 01:14:47 DEBUG "value": [ 01:14:47 DEBUG --- http response --- 01:14:47 DEBUG 01:14:47 DEBUG { 01:14:47 DEBUG 01:14:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:14:47 INFO [http_cmd]: http status code OK 01:14:47 DEBUG 01:14:47 DEBUG "metric": { 01:14:47 DEBUG 01:14:47 DEBUG ----- output ----- 01:14:47 DEBUG { 01:14:47 DEBUG [...] 01:14:47 DEBUG "pod": "ds-cts-0" 01:14:47 DEBUG 01:14:47 DEBUG [...] 01:14:47 DEBUG "status": "success", 01:14:47 DEBUG 01:14:47 DEBUG }, 01:14:47 DEBUG 01:14:47 DEBUG --- status code --- 01:14:47 DEBUG "data": { 01:14:47 DEBUG 01:14:47 DEBUG "value": [ 01:14:47 DEBUG 01:14:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:14:47 DEBUG "resultType": "vector", 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG --- http response --- 01:14:47 DEBUG "result": [ 01:14:47 DEBUG "pod": "ds-cts-2" 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG { 01:14:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:14:47 DEBUG }, 01:14:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:14:47 DEBUG [...] 01:14:47 DEBUG "metric": { 01:14:47 DEBUG ----- output ----- 01:14:47 DEBUG "value": [ 01:14:47 DEBUG }, 01:14:47 DEBUG 01:14:47 DEBUG "instance": "10.142.0.107:9100" 01:14:47 DEBUG { 01:14:47 DEBUG 1713485687, 01:14:47 DEBUG "value": [ 01:14:47 DEBUG 01:14:47 DEBUG }, 01:14:47 DEBUG "status": "success", 01:14:47 DEBUG "0.0002626266666666666" 01:14:47 DEBUG 1713485687, 01:14:47 DEBUG 01:14:47 DEBUG "value": [ 01:14:47 DEBUG "data": { 01:14:47 DEBUG "resultType": "vector", 01:14:47 DEBUG "868511744" 01:14:47 DEBUG "pod": "ds-cts-2" 01:14:47 DEBUG 01:14:47 DEBUG "result": [ 01:14:47 DEBUG ] 01:14:47 DEBUG ] 01:14:47 DEBUG }, 01:14:47 DEBUG 01:14:47 DEBUG { 01:14:47 DEBUG } 01:14:47 DEBUG } 01:14:47 DEBUG "value": [ 01:14:47 DEBUG 01:14:47 DEBUG "metric": { 01:14:47 DEBUG ] 01:14:47 DEBUG ] 01:14:47 DEBUG 1713485687, 01:14:47 DEBUG [...] 01:14:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:14:47 DEBUG } 01:14:47 DEBUG } 01:14:47 DEBUG "1767.1666666666667" 01:14:47 DEBUG 01:14:47 DEBUG }, 01:14:47 DEBUG } 01:14:47 DEBUG } 01:14:47 DEBUG ] 01:14:47 DEBUG 01:14:47 DEBUG "value": [ 01:14:47 DEBUG } 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG ] 01:14:47 DEBUG "instance": "10.142.0.43:9100" 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG }, 01:14:47 DEBUG [...] 01:14:47 DEBUG "value": [ 01:14:47 DEBUG } 01:14:47 DEBUG 01:14:47 DEBUG 1713485687, 01:14:47 DEBUG } 01:14:47 DEBUG 01:14:47 DEBUG 01:14:47 DEBUG "1.2416666666666647" 01:14:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:14:47 DEBUG ] 01:14:47 DEBUG }, 01:14:47 DEBUG } 01:14:47 DEBUG "value": [ 01:14:47 DEBUG ] 01:14:47 DEBUG 1713485687, 01:14:47 DEBUG } 01:14:47 DEBUG "20.076" 01:14:47 DEBUG } 01:14:47 DEBUG ] 01:14:47 DEBUG } 01:14:47 DEBUG ] 01:14:47 DEBUG } 01:14:47 DEBUG } 01:14:47 INFO 01:14:47 INFO [loop_until]: kubectl --namespace=xlou top pods 01:14:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:14:47 INFO [loop_until]: OK (rc = 0) 01:14:47 DEBUG --- stdout --- 01:14:47 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2805m 8965Mi am-6fd554f8f7-hvvkw 1807m 8897Mi am-6fd554f8f7-pmj4b 1997m 9077Mi ds-cts-0 8248m 12548Mi ds-cts-1 7739m 12531Mi ds-cts-2 9677m 12663Mi ds-idrepo-0 559m 3246Mi ds-idrepo-1 572m 3126Mi ds-idrepo-2 631m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 2m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 2159m 717Mi 01:14:47 DEBUG --- stderr --- 01:14:47 DEBUG 01:14:48 INFO 01:14:48 INFO [loop_until]: kubectl --namespace=xlou top node 01:14:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:14:48 INFO [loop_until]: OK (rc = 0) 01:14:48 DEBUG --- stdout --- 01:14:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5918Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 649m 2% 6647Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2096m 7% 11662Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3237Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8604m 28% 15204Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5223Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2647m 8% 11793Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 1036m 3% 3116Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2031m 6% 11337Mi 10% gke-xlou-cdm-ds-0f601751-j488 749m 2% 6104Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8060m 26% 16518Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 672m 2% 6759Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9047m 30% 14568Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 2695m 16% 4492Mi 7% 01:14:48 DEBUG --- stderr --- 01:14:48 DEBUG 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485746" 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485746" 01:15:46 ERROR [http_cmd]: ERROR 01:15:46 ERROR --- status code --- 01:15:46 INFO [http_cmd]: http status code OK 01:15:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:15:46 DEBUG --- status code --- 01:15:46 ERROR --- http response --- 01:15:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:46 DEBUG --- http response --- 01:15:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:15:46 DEBUG ----- output ----- 01:15:46 DEBUG { 01:15:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:15:46 DEBUG "status": "success", 01:15:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:15:46 DEBUG "data": { 01:15:46 DEBUG "resultType": "vector", 01:15:46 DEBUG "result": [ 01:15:46 DEBUG { 01:15:46 DEBUG "metric": { 01:15:46 DEBUG "instance": "10.142.0.107:9100" 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG [...] 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG "instance": "10.142.0.43:9100" 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 1713485746, 01:15:46 DEBUG "0.889766666666668" 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG } 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485746" 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485746" 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485746" 01:15:46 INFO [http_cmd]: http status code OK 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485746" 01:15:46 DEBUG --- status code --- 01:15:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:46 DEBUG --- http response --- 01:15:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:15:46 INFO [http_cmd]: http status code OK 01:15:46 DEBUG --- status code --- 01:15:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:46 DEBUG --- http response --- 01:15:46 INFO [http_cmd]: http status code OK 01:15:46 DEBUG --- status code --- 01:15:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:46 DEBUG --- http response --- 01:15:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:15:46 INFO [http_cmd]: http status code OK 01:15:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:46 DEBUG ----- output ----- 01:15:46 DEBUG --- status code --- 01:15:46 DEBUG ----- output ----- 01:15:46 DEBUG { 01:15:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:46 DEBUG { 01:15:46 DEBUG "status": "success", 01:15:46 DEBUG --- http response --- 01:15:46 DEBUG "status": "success", 01:15:46 DEBUG "data": { 01:15:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:15:46 DEBUG "data": { 01:15:46 DEBUG "resultType": "vector", 01:15:46 DEBUG ----- output ----- 01:15:46 DEBUG "resultType": "vector", 01:15:46 DEBUG "result": [ 01:15:46 DEBUG { 01:15:46 DEBUG "result": [ 01:15:46 DEBUG { 01:15:46 DEBUG "status": "success", 01:15:46 DEBUG { 01:15:46 DEBUG "metric": { 01:15:46 DEBUG "data": { 01:15:46 DEBUG "metric": { 01:15:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:15:46 DEBUG "resultType": "vector", 01:15:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:15:46 DEBUG }, 01:15:46 DEBUG "result": [ 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG { 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 01:15:46 DEBUG "metric": { 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG "instance": "10.142.0.107:9100" 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG [...] 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG [...] 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG [...] 01:15:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:15:46 DEBUG 01:15:46 DEBUG 1713485746, 01:15:46 DEBUG }, 01:15:46 DEBUG 01:15:46 DEBUG "555.3333559299055" 01:15:46 DEBUG "value": [ 01:15:46 DEBUG "instance": "10.142.0.43:9100" 01:15:46 DEBUG ] 01:15:46 DEBUG 1713485746, 01:15:46 DEBUG }, 01:15:46 DEBUG } 01:15:46 DEBUG "1891" 01:15:46 DEBUG "value": [ 01:15:46 DEBUG ] 01:15:46 DEBUG ] 01:15:46 DEBUG 1713485746, 01:15:46 DEBUG } 01:15:46 DEBUG } 01:15:46 DEBUG "346876222464" 01:15:46 DEBUG } 01:15:46 DEBUG ] 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG } 01:15:46 DEBUG } 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG } 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485746" 01:15:46 INFO [http_cmd]: http status code OK 01:15:46 DEBUG --- status code --- 01:15:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:46 DEBUG --- http response --- 01:15:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:15:46 DEBUG ----- output ----- 01:15:46 DEBUG { 01:15:46 DEBUG "status": "success", 01:15:46 DEBUG "data": { 01:15:46 DEBUG "resultType": "vector", 01:15:46 DEBUG "result": [ 01:15:46 DEBUG { 01:15:46 DEBUG "metric": { 01:15:46 DEBUG "pod": "ds-idrepo-0" 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG [...] 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG "pod": "ds-cts-2" 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 1713485746, 01:15:46 DEBUG "0" 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG } 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485746" 01:15:46 INFO [http_cmd]: http status code OK 01:15:46 DEBUG --- status code --- 01:15:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:46 DEBUG --- http response --- 01:15:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:15:46 DEBUG ----- output ----- 01:15:46 DEBUG { 01:15:46 DEBUG "status": "success", 01:15:46 DEBUG "data": { 01:15:46 DEBUG "resultType": "vector", 01:15:46 DEBUG "result": [ 01:15:46 DEBUG { 01:15:46 DEBUG "metric": { 01:15:46 DEBUG "instance": "10.142.0.107:9100" 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG [...] 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG "instance": "10.142.0.43:9100" 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 1713485746, 01:15:46 DEBUG "38700990.43333334" 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG } 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485746" 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485746" 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485746" 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485746" 01:15:46 INFO [http_cmd]: http status code OK 01:15:46 DEBUG --- status code --- 01:15:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:46 DEBUG --- http response --- 01:15:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:15:46 DEBUG ----- output ----- 01:15:46 DEBUG { 01:15:46 DEBUG "status": "success", 01:15:46 DEBUG "data": { 01:15:46 DEBUG "resultType": "vector", 01:15:46 DEBUG "result": [ 01:15:46 DEBUG { 01:15:46 DEBUG "metric": { 01:15:46 DEBUG "pod": "ds-cts-0" 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG [...] 01:15:46 INFO [http_cmd]: http status code OK 01:15:46 DEBUG 01:15:46 DEBUG --- status code --- 01:15:46 DEBUG 01:15:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:46 DEBUG 01:15:46 DEBUG --- http response --- 01:15:46 DEBUG "pod": "ds-cts-2" 01:15:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:46 DEBUG }, 01:15:46 DEBUG ----- output ----- 01:15:46 DEBUG "value": [ 01:15:46 DEBUG { 01:15:46 DEBUG 1713485746, 01:15:46 DEBUG "status": "success", 01:15:46 DEBUG "0" 01:15:46 DEBUG "data": { 01:15:46 DEBUG ] 01:15:46 DEBUG "resultType": "vector", 01:15:46 DEBUG } 01:15:46 DEBUG "result": [ 01:15:46 INFO [http_cmd]: http status code OK 01:15:46 DEBUG ] 01:15:46 DEBUG { 01:15:46 DEBUG --- status code --- 01:15:46 DEBUG } 01:15:46 DEBUG "metric": { 01:15:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:46 DEBUG } 01:15:46 DEBUG "pod": "ds-idrepo-0" 01:15:46 DEBUG --- http response --- 01:15:46 DEBUG }, 01:15:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:46 DEBUG "value": [ 01:15:46 DEBUG ----- output ----- 01:15:46 DEBUG 01:15:46 DEBUG { 01:15:46 DEBUG "status": "success", 01:15:46 DEBUG 01:15:46 DEBUG "data": { 01:15:46 DEBUG 01:15:46 DEBUG "resultType": "vector", 01:15:46 DEBUG [...] 01:15:46 DEBUG "result": [ 01:15:46 DEBUG 01:15:46 DEBUG { 01:15:46 DEBUG 01:15:46 DEBUG "metric": { 01:15:46 DEBUG 01:15:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:15:46 DEBUG "pod": "ds-idrepo-2" 01:15:46 DEBUG }, 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG "value": [ 01:15:46 INFO [http_cmd]: http status code OK 01:15:46 DEBUG 01:15:46 DEBUG 1713485746, 01:15:46 DEBUG --- status code --- 01:15:46 DEBUG 01:15:46 DEBUG "0" 01:15:46 DEBUG ] 01:15:46 DEBUG 01:15:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:46 DEBUG } 01:15:46 DEBUG [...] 01:15:46 DEBUG --- http response --- 01:15:46 DEBUG ] 01:15:46 DEBUG 01:15:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:15:46 DEBUG } 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG } 01:15:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:15:46 DEBUG ----- output ----- 01:15:46 DEBUG }, 01:15:46 DEBUG { 01:15:46 DEBUG "status": "success", 01:15:46 DEBUG "data": { 01:15:46 DEBUG "value": [ 01:15:46 DEBUG "resultType": "vector", 01:15:46 DEBUG 1713485746, 01:15:46 DEBUG "result": [ 01:15:46 DEBUG "1150.3" 01:15:46 DEBUG { 01:15:46 DEBUG ] 01:15:46 DEBUG "metric": { 01:15:46 DEBUG } 01:15:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:15:46 DEBUG ] 01:15:46 DEBUG }, 01:15:46 DEBUG } 01:15:46 DEBUG "value": [ 01:15:46 DEBUG } 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG [...] 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 1713485746, 01:15:46 DEBUG "316440330.54265666" 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG } 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485746" 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485746" 01:15:46 INFO [http_cmd]: http status code OK 01:15:46 INFO [http_cmd]: http status code OK 01:15:46 DEBUG --- status code --- 01:15:46 DEBUG --- status code --- 01:15:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:46 DEBUG --- http response --- 01:15:46 DEBUG --- http response --- 01:15:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:46 DEBUG ----- output ----- 01:15:46 DEBUG { 01:15:46 DEBUG "status": "success", 01:15:46 DEBUG "data": { 01:15:46 DEBUG "resultType": "vector", 01:15:46 DEBUG "result": [ 01:15:46 DEBUG { 01:15:46 DEBUG "metric": { 01:15:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG [...] 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 1713485746, 01:15:46 DEBUG "0" 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG } 01:15:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:15:46 DEBUG ----- output ----- 01:15:46 DEBUG { 01:15:46 DEBUG "status": "success", 01:15:46 DEBUG "data": { 01:15:46 DEBUG "resultType": "vector", 01:15:46 DEBUG "result": [ 01:15:46 DEBUG { 01:15:46 DEBUG "metric": { 01:15:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG [...] 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 1713485746, 01:15:46 DEBUG "8.033364320654174" 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG } 01:15:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485746" 01:15:46 INFO [http_cmd]: http status code OK 01:15:46 DEBUG --- status code --- 01:15:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:46 DEBUG --- http response --- 01:15:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:15:46 DEBUG ----- output ----- 01:15:46 DEBUG { 01:15:46 DEBUG "status": "success", 01:15:46 DEBUG "data": { 01:15:46 DEBUG "resultType": "vector", 01:15:46 DEBUG "result": [ 01:15:46 DEBUG { 01:15:46 DEBUG "metric": { 01:15:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG [...] 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG 01:15:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:15:46 DEBUG }, 01:15:46 DEBUG "value": [ 01:15:46 DEBUG 1713485746, 01:15:46 DEBUG "8.47988882486534" 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG ] 01:15:46 DEBUG } 01:15:46 DEBUG } 01:15:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485747" 01:15:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485747" 01:15:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485747" 01:15:47 INFO [http_cmd]: http status code OK 01:15:47 DEBUG --- status code --- 01:15:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:47 DEBUG --- http response --- 01:15:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:15:47 DEBUG ----- output ----- 01:15:47 DEBUG { 01:15:47 DEBUG "status": "success", 01:15:47 DEBUG "data": { 01:15:47 DEBUG "resultType": "vector", 01:15:47 DEBUG "result": [ 01:15:47 DEBUG { 01:15:47 DEBUG "metric": { 01:15:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG [...] 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 1713485747, 01:15:47 DEBUG "0" 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG } 01:15:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485747" 01:15:47 INFO [http_cmd]: http status code OK 01:15:47 INFO [http_cmd]: http status code OK 01:15:47 DEBUG --- status code --- 01:15:47 DEBUG --- status code --- 01:15:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:47 DEBUG --- http response --- 01:15:47 DEBUG --- http response --- 01:15:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:47 DEBUG ----- output ----- 01:15:47 DEBUG { 01:15:47 DEBUG "status": "success", 01:15:47 DEBUG "data": { 01:15:47 DEBUG "resultType": "vector", 01:15:47 DEBUG "result": [ 01:15:47 DEBUG { 01:15:47 DEBUG "metric": { 01:15:47 DEBUG "pod": "ds-idrepo-0" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 01:15:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:47 DEBUG 01:15:47 DEBUG ----- output ----- 01:15:47 DEBUG 01:15:47 DEBUG { 01:15:47 DEBUG [...] 01:15:47 DEBUG "status": "success", 01:15:47 DEBUG 01:15:47 DEBUG "data": { 01:15:47 DEBUG 01:15:47 DEBUG "resultType": "vector", 01:15:47 DEBUG 01:15:47 DEBUG "result": [ 01:15:47 DEBUG "pod": "ds-idrepo-2" 01:15:47 DEBUG { 01:15:47 DEBUG }, 01:15:47 DEBUG "metric": { 01:15:47 DEBUG "value": [ 01:15:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 1713485747, 01:15:47 DEBUG 01:15:47 DEBUG "0" 01:15:47 DEBUG 01:15:47 DEBUG ] 01:15:47 DEBUG 01:15:47 DEBUG } 01:15:47 DEBUG [...] 01:15:47 DEBUG ] 01:15:47 DEBUG 01:15:47 DEBUG } 01:15:47 DEBUG 01:15:47 DEBUG } 01:15:47 DEBUG 01:15:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 1713485747, 01:15:47 DEBUG "1151.2666666666667" 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG } 01:15:47 INFO [http_cmd]: http status code OK 01:15:47 DEBUG --- status code --- 01:15:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:47 DEBUG --- http response --- 01:15:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:47 DEBUG ----- output ----- 01:15:47 DEBUG { 01:15:47 DEBUG "status": "success", 01:15:47 DEBUG "data": { 01:15:47 DEBUG "resultType": "vector", 01:15:47 DEBUG "result": [ 01:15:47 DEBUG { 01:15:47 DEBUG "metric": { 01:15:47 DEBUG "pod": "ds-idrepo-0" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG [...] 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG "pod": "ds-idrepo-2" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 1713485747, 01:15:47 DEBUG "0" 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG } 01:15:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485747" 01:15:47 INFO [http_cmd]: http status code OK 01:15:47 DEBUG --- status code --- 01:15:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:47 DEBUG --- http response --- 01:15:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:47 DEBUG ----- output ----- 01:15:47 DEBUG { 01:15:47 DEBUG "status": "success", 01:15:47 DEBUG "data": { 01:15:47 DEBUG "resultType": "vector", 01:15:47 DEBUG "result": [ 01:15:47 DEBUG { 01:15:47 DEBUG "metric": { 01:15:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG [...] 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 1713485747, 01:15:47 DEBUG "1150.8666666666666" 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG } 01:15:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485747" 01:15:47 INFO [http_cmd]: http status code OK 01:15:47 DEBUG --- status code --- 01:15:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:47 DEBUG --- http response --- 01:15:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:47 DEBUG ----- output ----- 01:15:47 DEBUG { 01:15:47 DEBUG "status": "success", 01:15:47 DEBUG "data": { 01:15:47 DEBUG "resultType": "vector", 01:15:47 DEBUG "result": [ 01:15:47 DEBUG { 01:15:47 DEBUG "metric": { 01:15:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG [...] 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 1713485747, 01:15:47 DEBUG "3327390" 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG } 01:15:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485747" 01:15:47 INFO [http_cmd]: http status code OK 01:15:47 DEBUG --- status code --- 01:15:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:47 DEBUG --- http response --- 01:15:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:15:47 DEBUG ----- output ----- 01:15:47 DEBUG { 01:15:47 DEBUG "status": "success", 01:15:47 DEBUG "data": { 01:15:47 DEBUG "resultType": "vector", 01:15:47 DEBUG "result": [ 01:15:47 DEBUG { 01:15:47 DEBUG "metric": { 01:15:47 DEBUG "instance": "10.142.0.107:9100" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG [...] 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG "instance": "10.142.0.43:9100" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 1713485747, 01:15:47 DEBUG "38706435.96666667" 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG } 01:15:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485747" 01:15:47 INFO [http_cmd]: http status code OK 01:15:47 DEBUG --- status code --- 01:15:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:47 DEBUG --- http response --- 01:15:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:15:47 DEBUG ----- output ----- 01:15:47 DEBUG { 01:15:47 DEBUG "status": "success", 01:15:47 DEBUG "data": { 01:15:47 DEBUG "resultType": "vector", 01:15:47 DEBUG "result": [ 01:15:47 DEBUG { 01:15:47 DEBUG "metric": { 01:15:47 DEBUG "pod": "ds-cts-0" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG [...] 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG "pod": "ds-cts-2" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 1713485747, 01:15:47 DEBUG "0.0002563133333333333" 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG } 01:15:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485747" 01:15:47 INFO [http_cmd]: http status code OK 01:15:47 DEBUG --- status code --- 01:15:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:47 DEBUG --- http response --- 01:15:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:15:47 DEBUG ----- output ----- 01:15:47 DEBUG { 01:15:47 DEBUG "status": "success", 01:15:47 DEBUG "data": { 01:15:47 DEBUG "resultType": "vector", 01:15:47 DEBUG "result": [ 01:15:47 DEBUG { 01:15:47 DEBUG "metric": { 01:15:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG [...] 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 1713485747, 01:15:47 DEBUG "20.076" 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG } 01:15:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485747" 01:15:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485747" 01:15:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485747" 01:15:47 INFO [http_cmd]: http status code OK 01:15:47 DEBUG --- status code --- 01:15:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:47 DEBUG --- http response --- 01:15:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:15:47 DEBUG ----- output ----- 01:15:47 DEBUG { 01:15:47 DEBUG "status": "success", 01:15:47 DEBUG "data": { 01:15:47 DEBUG "resultType": "vector", 01:15:47 DEBUG "result": [ 01:15:47 DEBUG { 01:15:47 DEBUG "metric": { 01:15:47 DEBUG "pod": "ds-cts-0" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG [...] 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG "pod": "ds-cts-2" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 1713485747, 01:15:47 DEBUG "1941.2333333333333" 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG } 01:15:47 INFO [http_cmd]: http status code OK 01:15:47 DEBUG --- status code --- 01:15:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:47 DEBUG --- http response --- 01:15:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:15:47 DEBUG ----- output ----- 01:15:47 DEBUG { 01:15:47 DEBUG "status": "success", 01:15:47 DEBUG "data": { 01:15:47 DEBUG "resultType": "vector", 01:15:47 DEBUG "result": [ 01:15:47 DEBUG { 01:15:47 DEBUG "metric": { 01:15:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG [...] 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 1713485747, 01:15:47 DEBUG "3281960960" 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG } 01:15:47 INFO [http_cmd]: http status code OK 01:15:47 DEBUG --- status code --- 01:15:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:15:47 DEBUG --- http response --- 01:15:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:15:47 DEBUG ----- output ----- 01:15:47 DEBUG { 01:15:47 DEBUG "status": "success", 01:15:47 DEBUG "data": { 01:15:47 DEBUG "resultType": "vector", 01:15:47 DEBUG "result": [ 01:15:47 DEBUG { 01:15:47 DEBUG "metric": { 01:15:47 DEBUG "instance": "10.142.0.107:9100" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG [...] 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG 01:15:47 DEBUG "instance": "10.142.0.43:9100" 01:15:47 DEBUG }, 01:15:47 DEBUG "value": [ 01:15:47 DEBUG 1713485747, 01:15:47 DEBUG "0.9620000000000003" 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG ] 01:15:47 DEBUG } 01:15:47 DEBUG } 01:15:47 INFO 01:15:47 INFO [loop_until]: kubectl --namespace=xlou top pods 01:15:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:15:48 INFO [loop_until]: OK (rc = 0) 01:15:48 DEBUG --- stdout --- 01:15:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 10596m 8973Mi am-6fd554f8f7-hvvkw 8155m 8901Mi am-6fd554f8f7-pmj4b 9995m 9079Mi ds-cts-0 16373m 12534Mi ds-cts-1 15102m 12349Mi ds-cts-2 16072m 12450Mi ds-idrepo-0 4054m 3249Mi ds-idrepo-1 4320m 3163Mi ds-idrepo-2 4201m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 14m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7609m 2062Mi 01:15:48 DEBUG --- stderr --- 01:15:48 DEBUG 01:15:48 INFO 01:15:48 INFO [loop_until]: kubectl --namespace=xlou top node 01:15:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:15:48 INFO [loop_until]: OK (rc = 0) 01:15:48 DEBUG --- stdout --- 01:15:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5928Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4476m 14% 6662Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8360m 28% 11641Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3237Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 16383m 54% 14979Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 5219Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10560m 35% 11776Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 19445m 65% 3895Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 10537m 35% 11298Mi 10% gke-xlou-cdm-ds-0f601751-j488 4332m 14% 6094Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 14623m 48% 16332Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 4791m 16% 6793Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 16141m 54% 14309Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 7872m 49% 6520Mi 11% 01:15:48 DEBUG --- stderr --- 01:15:48 DEBUG 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485806" 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485806" 01:16:46 ERROR [http_cmd]: ERROR 01:16:46 ERROR --- status code --- 01:16:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:16:46 ERROR --- http response --- 01:16:46 INFO [http_cmd]: http status code OK 01:16:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:16:46 DEBUG --- status code --- 01:16:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:16:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:46 DEBUG --- http response --- 01:16:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:46 DEBUG ----- output ----- 01:16:46 DEBUG { 01:16:46 DEBUG "status": "success", 01:16:46 DEBUG "data": { 01:16:46 DEBUG "resultType": "vector", 01:16:46 DEBUG "result": [ 01:16:46 DEBUG { 01:16:46 DEBUG "metric": { 01:16:46 DEBUG "instance": "10.142.0.107:9100" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG [...] 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG "instance": "10.142.0.43:9100" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 1713485806, 01:16:46 DEBUG "0.8836666666666683" 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG } 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485806" 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485806" 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485806" 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485806" 01:16:46 INFO [http_cmd]: http status code OK 01:16:46 DEBUG --- status code --- 01:16:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:46 DEBUG --- http response --- 01:16:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:16:46 INFO [http_cmd]: http status code OK 01:16:46 DEBUG --- status code --- 01:16:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:46 DEBUG --- http response --- 01:16:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:46 DEBUG ----- output ----- 01:16:46 DEBUG { 01:16:46 DEBUG "status": "success", 01:16:46 DEBUG "data": { 01:16:46 DEBUG "resultType": "vector", 01:16:46 DEBUG "result": [ 01:16:46 DEBUG { 01:16:46 DEBUG "metric": { 01:16:46 DEBUG "instance": "10.142.0.107:9100" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG [...] 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG "instance": "10.142.0.43:9100" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 1713485806, 01:16:46 DEBUG "348810892288" 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG } 01:16:46 INFO [http_cmd]: http status code OK 01:16:46 INFO [http_cmd]: http status code OK 01:16:46 DEBUG --- status code --- 01:16:46 DEBUG --- status code --- 01:16:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:46 DEBUG --- http response --- 01:16:46 DEBUG --- http response --- 01:16:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:16:46 DEBUG ----- output ----- 01:16:46 DEBUG ----- output ----- 01:16:46 DEBUG { 01:16:46 DEBUG { 01:16:46 DEBUG "status": "success", 01:16:46 DEBUG "status": "success", 01:16:46 DEBUG "data": { 01:16:46 DEBUG "data": { 01:16:46 DEBUG "resultType": "vector", 01:16:46 DEBUG "resultType": "vector", 01:16:46 DEBUG "result": [ 01:16:46 DEBUG "result": [ 01:16:46 DEBUG { 01:16:46 DEBUG { 01:16:46 DEBUG "metric": { 01:16:46 DEBUG "metric": { 01:16:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:16:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:16:46 DEBUG }, 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG [...] 01:16:46 DEBUG [...] 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:16:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:16:46 DEBUG }, 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 1713485806, 01:16:46 DEBUG 1713485806, 01:16:46 DEBUG "1899.0666666666666" 01:16:46 DEBUG "567.9974869306328" 01:16:46 DEBUG ] 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG } 01:16:46 DEBUG ] 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG } 01:16:46 DEBUG } 01:16:46 DEBUG } 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485806" 01:16:46 INFO [http_cmd]: http status code OK 01:16:46 DEBUG --- status code --- 01:16:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:46 DEBUG --- http response --- 01:16:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:16:46 DEBUG ----- output ----- 01:16:46 DEBUG { 01:16:46 DEBUG "status": "success", 01:16:46 DEBUG "data": { 01:16:46 DEBUG "resultType": "vector", 01:16:46 DEBUG "result": [ 01:16:46 DEBUG { 01:16:46 DEBUG "metric": { 01:16:46 DEBUG "pod": "ds-idrepo-0" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG [...] 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG "pod": "ds-cts-2" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 1713485806, 01:16:46 DEBUG "0" 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG } 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485806" 01:16:46 INFO [http_cmd]: http status code OK 01:16:46 DEBUG --- status code --- 01:16:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:46 DEBUG --- http response --- 01:16:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:46 DEBUG ----- output ----- 01:16:46 DEBUG { 01:16:46 DEBUG "status": "success", 01:16:46 DEBUG "data": { 01:16:46 DEBUG "resultType": "vector", 01:16:46 DEBUG "result": [ 01:16:46 DEBUG { 01:16:46 DEBUG "metric": { 01:16:46 DEBUG "instance": "10.142.0.107:9100" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG [...] 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG "instance": "10.142.0.43:9100" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 1713485806, 01:16:46 DEBUG "40055543.63333333" 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG } 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485806" 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485806" 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485806" 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485806" 01:16:46 INFO [http_cmd]: http status code OK 01:16:46 DEBUG --- status code --- 01:16:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:46 DEBUG --- http response --- 01:16:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:16:46 DEBUG ----- output ----- 01:16:46 DEBUG { 01:16:46 DEBUG "status": "success", 01:16:46 DEBUG "data": { 01:16:46 DEBUG "resultType": "vector", 01:16:46 DEBUG "result": [ 01:16:46 DEBUG { 01:16:46 DEBUG "metric": { 01:16:46 DEBUG "pod": "ds-cts-0" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG [...] 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG "pod": "ds-cts-2" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 1713485806, 01:16:46 DEBUG "0" 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG } 01:16:46 INFO [http_cmd]: http status code OK 01:16:46 DEBUG --- status code --- 01:16:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:46 DEBUG --- http response --- 01:16:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:46 DEBUG ----- output ----- 01:16:46 DEBUG { 01:16:46 DEBUG "status": "success", 01:16:46 DEBUG "data": { 01:16:46 DEBUG "resultType": "vector", 01:16:46 DEBUG "result": [ 01:16:46 DEBUG { 01:16:46 DEBUG "metric": { 01:16:46 DEBUG "pod": "ds-idrepo-0" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG [...] 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG "pod": "ds-idrepo-2" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 1713485806, 01:16:46 DEBUG "0" 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG } 01:16:46 INFO [http_cmd]: http status code OK 01:16:46 DEBUG --- status code --- 01:16:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:46 DEBUG --- http response --- 01:16:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:46 INFO [http_cmd]: http status code OK 01:16:46 DEBUG ----- output ----- 01:16:46 DEBUG --- status code --- 01:16:46 DEBUG { 01:16:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:46 DEBUG "status": "success", 01:16:46 DEBUG --- http response --- 01:16:46 DEBUG "data": { 01:16:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:16:46 DEBUG "resultType": "vector", 01:16:46 DEBUG ----- output ----- 01:16:46 DEBUG "result": [ 01:16:46 DEBUG { 01:16:46 DEBUG { 01:16:46 DEBUG "status": "success", 01:16:46 DEBUG "metric": { 01:16:46 DEBUG "data": { 01:16:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:16:46 DEBUG "resultType": "vector", 01:16:46 DEBUG }, 01:16:46 DEBUG "result": [ 01:16:46 DEBUG "value": [ 01:16:46 DEBUG { 01:16:46 DEBUG 01:16:46 DEBUG "metric": { 01:16:46 DEBUG 01:16:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:16:46 DEBUG 01:16:46 DEBUG }, 01:16:46 DEBUG [...] 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:16:46 DEBUG [...] 01:16:46 DEBUG }, 01:16:46 DEBUG 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 01:16:46 DEBUG 1713485806, 01:16:46 DEBUG 01:16:46 DEBUG "1089" 01:16:46 DEBUG "pod": "ds-cts-2" 01:16:46 DEBUG ] 01:16:46 DEBUG }, 01:16:46 DEBUG } 01:16:46 DEBUG "value": [ 01:16:46 DEBUG ] 01:16:46 DEBUG 1713485806, 01:16:46 DEBUG } 01:16:46 DEBUG "0" 01:16:46 DEBUG } 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG } 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485806" 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485806" 01:16:46 INFO [http_cmd]: http status code OK 01:16:46 DEBUG --- status code --- 01:16:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:46 DEBUG --- http response --- 01:16:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:46 DEBUG ----- output ----- 01:16:46 DEBUG { 01:16:46 DEBUG "status": "success", 01:16:46 DEBUG "data": { 01:16:46 DEBUG "resultType": "vector", 01:16:46 DEBUG "result": [ 01:16:46 DEBUG { 01:16:46 DEBUG "metric": { 01:16:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG [...] 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 1713485806, 01:16:46 DEBUG "0" 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG } 01:16:46 INFO [http_cmd]: http status code OK 01:16:46 DEBUG --- status code --- 01:16:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:46 DEBUG --- http response --- 01:16:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:46 DEBUG ----- output ----- 01:16:46 DEBUG { 01:16:46 DEBUG "status": "success", 01:16:46 DEBUG "data": { 01:16:46 DEBUG "resultType": "vector", 01:16:46 DEBUG "result": [ 01:16:46 DEBUG { 01:16:46 DEBUG "metric": { 01:16:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG [...] 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 1713485806, 01:16:46 DEBUG "8.032488923890458" 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG } 01:16:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485806" 01:16:46 INFO [http_cmd]: http status code OK 01:16:46 DEBUG --- status code --- 01:16:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:46 DEBUG --- http response --- 01:16:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:16:46 DEBUG ----- output ----- 01:16:46 DEBUG { 01:16:46 DEBUG "status": "success", 01:16:46 DEBUG "data": { 01:16:46 DEBUG "resultType": "vector", 01:16:46 DEBUG "result": [ 01:16:46 DEBUG { 01:16:46 DEBUG "metric": { 01:16:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG [...] 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG 01:16:46 DEBUG "pod": "ds-cts-2" 01:16:46 DEBUG }, 01:16:46 DEBUG "value": [ 01:16:46 DEBUG 1713485806, 01:16:46 DEBUG "0" 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG ] 01:16:46 DEBUG } 01:16:46 DEBUG } 01:16:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485807" 01:16:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485807" 01:16:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485807" 01:16:47 INFO [http_cmd]: http status code OK 01:16:47 INFO [http_cmd]: http status code OK 01:16:47 DEBUG --- status code --- 01:16:47 DEBUG --- status code --- 01:16:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:47 DEBUG --- http response --- 01:16:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485807" 01:16:47 DEBUG --- http response --- 01:16:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:47 DEBUG ----- output ----- 01:16:47 DEBUG ----- output ----- 01:16:47 DEBUG { 01:16:47 DEBUG { 01:16:47 DEBUG "status": "success", 01:16:47 DEBUG "status": "success", 01:16:47 DEBUG "data": { 01:16:47 DEBUG "data": { 01:16:47 DEBUG "resultType": "vector", 01:16:47 DEBUG "resultType": "vector", 01:16:47 DEBUG "result": [ 01:16:47 DEBUG "result": [ 01:16:47 DEBUG { 01:16:47 DEBUG { 01:16:47 DEBUG "metric": { 01:16:47 DEBUG "metric": { 01:16:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:16:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:16:47 DEBUG }, 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG [...] 01:16:47 DEBUG [...] 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:16:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:16:47 DEBUG }, 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 1713485807, 01:16:47 DEBUG 1713485807, 01:16:47 DEBUG "1087.3" 01:16:47 DEBUG "0" 01:16:47 DEBUG ] 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG } 01:16:47 DEBUG ] 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG } 01:16:47 DEBUG } 01:16:47 DEBUG } 01:16:47 INFO [http_cmd]: http status code OK 01:16:47 DEBUG --- status code --- 01:16:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:47 DEBUG --- http response --- 01:16:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:47 DEBUG ----- output ----- 01:16:47 DEBUG { 01:16:47 DEBUG "status": "success", 01:16:47 DEBUG "data": { 01:16:47 DEBUG "resultType": "vector", 01:16:47 DEBUG "result": [ 01:16:47 DEBUG { 01:16:47 DEBUG "metric": { 01:16:47 DEBUG "pod": "ds-idrepo-0" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG [...] 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG "pod": "ds-idrepo-2" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 1713485807, 01:16:47 DEBUG "0" 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG } 01:16:47 INFO [http_cmd]: http status code OK 01:16:47 DEBUG --- status code --- 01:16:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:47 DEBUG --- http response --- 01:16:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:47 DEBUG ----- output ----- 01:16:47 DEBUG { 01:16:47 DEBUG "status": "success", 01:16:47 DEBUG "data": { 01:16:47 DEBUG "resultType": "vector", 01:16:47 DEBUG "result": [ 01:16:47 DEBUG { 01:16:47 DEBUG "metric": { 01:16:47 DEBUG "pod": "ds-idrepo-0" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG [...] 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG "pod": "ds-idrepo-2" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 1713485807, 01:16:47 DEBUG "0" 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG } 01:16:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485807" 01:16:47 INFO [http_cmd]: http status code OK 01:16:47 DEBUG --- status code --- 01:16:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:47 DEBUG --- http response --- 01:16:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:47 DEBUG ----- output ----- 01:16:47 DEBUG { 01:16:47 DEBUG "status": "success", 01:16:47 DEBUG "data": { 01:16:47 DEBUG "resultType": "vector", 01:16:47 DEBUG "result": [ 01:16:47 DEBUG { 01:16:47 DEBUG "metric": { 01:16:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG [...] 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 1713485807, 01:16:47 DEBUG "1088.2666666666667" 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG } 01:16:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485807" 01:16:47 INFO [http_cmd]: http status code OK 01:16:47 DEBUG --- status code --- 01:16:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:47 DEBUG --- http response --- 01:16:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:47 DEBUG ----- output ----- 01:16:47 DEBUG { 01:16:47 DEBUG "status": "success", 01:16:47 DEBUG "data": { 01:16:47 DEBUG "resultType": "vector", 01:16:47 DEBUG "result": [ 01:16:47 DEBUG { 01:16:47 DEBUG "metric": { 01:16:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG [...] 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 1713485807, 01:16:47 DEBUG "3280832" 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG } 01:16:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485807" 01:16:47 INFO [http_cmd]: http status code OK 01:16:47 DEBUG --- status code --- 01:16:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:47 DEBUG --- http response --- 01:16:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:47 DEBUG ----- output ----- 01:16:47 DEBUG { 01:16:47 DEBUG "status": "success", 01:16:47 DEBUG "data": { 01:16:47 DEBUG "resultType": "vector", 01:16:47 DEBUG "result": [ 01:16:47 DEBUG { 01:16:47 DEBUG "metric": { 01:16:47 DEBUG "instance": "10.142.0.107:9100" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG [...] 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485807" 01:16:47 DEBUG 01:16:47 DEBUG "instance": "10.142.0.43:9100" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 1713485807, 01:16:47 DEBUG "40058121.8" 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG } 01:16:47 INFO [http_cmd]: http status code OK 01:16:47 DEBUG --- status code --- 01:16:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:47 DEBUG --- http response --- 01:16:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:16:47 DEBUG ----- output ----- 01:16:47 DEBUG { 01:16:47 DEBUG "status": "success", 01:16:47 DEBUG "data": { 01:16:47 DEBUG "resultType": "vector", 01:16:47 DEBUG "result": [ 01:16:47 DEBUG { 01:16:47 DEBUG "metric": { 01:16:47 DEBUG "pod": "ds-cts-0" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG [...] 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG "pod": "ds-cts-2" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 1713485807, 01:16:47 DEBUG "0.00023333333333333333" 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG } 01:16:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485807" 01:16:47 INFO [http_cmd]: http status code OK 01:16:47 DEBUG --- status code --- 01:16:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:47 DEBUG --- http response --- 01:16:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:16:47 DEBUG ----- output ----- 01:16:47 DEBUG { 01:16:47 DEBUG "status": "success", 01:16:47 DEBUG "data": { 01:16:47 DEBUG "resultType": "vector", 01:16:47 DEBUG "result": [ 01:16:47 DEBUG { 01:16:47 DEBUG "metric": { 01:16:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG [...] 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 1713485807, 01:16:47 DEBUG "20.076" 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG } 01:16:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485807" 01:16:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485807" 01:16:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485807" 01:16:47 INFO [http_cmd]: http status code OK 01:16:47 DEBUG --- status code --- 01:16:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:47 DEBUG --- http response --- 01:16:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:16:47 DEBUG ----- output ----- 01:16:47 DEBUG { 01:16:47 DEBUG "status": "success", 01:16:47 DEBUG "data": { 01:16:47 DEBUG "resultType": "vector", 01:16:47 DEBUG "result": [ 01:16:47 DEBUG { 01:16:47 DEBUG "metric": { 01:16:47 DEBUG "pod": "ds-cts-0" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG [...] 01:16:47 DEBUG 01:16:47 INFO [http_cmd]: http status code OK 01:16:47 DEBUG 01:16:47 DEBUG --- status code --- 01:16:47 DEBUG 01:16:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:47 DEBUG "pod": "ds-cts-2" 01:16:47 DEBUG --- http response --- 01:16:47 DEBUG }, 01:16:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:47 DEBUG "value": [ 01:16:47 DEBUG ----- output ----- 01:16:47 DEBUG 1713485807, 01:16:47 DEBUG { 01:16:47 DEBUG "378.4" 01:16:47 DEBUG "status": "success", 01:16:47 DEBUG ] 01:16:47 DEBUG "data": { 01:16:47 DEBUG } 01:16:47 DEBUG "resultType": "vector", 01:16:47 DEBUG ] 01:16:47 DEBUG "result": [ 01:16:47 DEBUG } 01:16:47 DEBUG { 01:16:47 INFO [http_cmd]: http status code OK 01:16:47 DEBUG } 01:16:47 DEBUG "metric": { 01:16:47 DEBUG --- status code --- 01:16:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:16:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:16:47 DEBUG }, 01:16:47 DEBUG --- http response --- 01:16:47 DEBUG "value": [ 01:16:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:16:47 DEBUG 01:16:47 DEBUG ----- output ----- 01:16:47 DEBUG 01:16:47 DEBUG { 01:16:47 DEBUG 01:16:47 DEBUG "status": "success", 01:16:47 DEBUG [...] 01:16:47 DEBUG "data": { 01:16:47 DEBUG 01:16:47 DEBUG "resultType": "vector", 01:16:47 DEBUG 01:16:47 DEBUG "result": [ 01:16:47 DEBUG 01:16:47 DEBUG { 01:16:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:16:47 DEBUG "metric": { 01:16:47 DEBUG }, 01:16:47 DEBUG "instance": "10.142.0.107:9100" 01:16:47 DEBUG "value": [ 01:16:47 DEBUG }, 01:16:47 DEBUG 1713485807, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG "4918820864" 01:16:47 DEBUG 01:16:47 DEBUG ] 01:16:47 DEBUG 01:16:47 DEBUG } 01:16:47 DEBUG 01:16:47 DEBUG ] 01:16:47 DEBUG [...] 01:16:47 DEBUG } 01:16:47 DEBUG 01:16:47 DEBUG } 01:16:47 DEBUG 01:16:47 DEBUG 01:16:47 DEBUG "instance": "10.142.0.43:9100" 01:16:47 DEBUG }, 01:16:47 DEBUG "value": [ 01:16:47 DEBUG 1713485807, 01:16:47 DEBUG "1.0766666666666662" 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG ] 01:16:47 DEBUG } 01:16:47 DEBUG } 01:16:48 INFO 01:16:48 INFO [loop_until]: kubectl --namespace=xlou top pods 01:16:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:16:48 INFO [loop_until]: OK (rc = 0) 01:16:48 DEBUG --- stdout --- 01:16:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 8728m 8972Mi am-6fd554f8f7-hvvkw 7088m 8906Mi am-6fd554f8f7-pmj4b 7633m 9078Mi ds-cts-0 14153m 12439Mi ds-cts-1 12071m 12476Mi ds-cts-2 13240m 12504Mi ds-idrepo-0 3384m 3249Mi ds-idrepo-1 3256m 3167Mi ds-idrepo-2 3376m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 4m 1215Mi lodemon-5d46798758-5h2st 12m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8058m 4596Mi 01:16:48 DEBUG --- stderr --- 01:16:48 DEBUG 01:16:48 INFO 01:16:48 INFO [loop_until]: kubectl --namespace=xlou top node 01:16:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:16:48 INFO [loop_until]: OK (rc = 0) 01:16:48 DEBUG --- stdout --- 01:16:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5925Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3397m 11% 6658Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6871m 23% 11637Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 85m 0% 3231Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 14192m 47% 14945Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5202Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8713m 29% 11774Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 15612m 52% 3896Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 8177m 27% 11303Mi 10% gke-xlou-cdm-ds-0f601751-j488 3353m 11% 6088Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 12791m 42% 16416Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3654m 12% 6795Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 13805m 46% 14353Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 8188m 51% 8497Mi 14% 01:16:48 DEBUG --- stderr --- 01:16:48 DEBUG 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485866" 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485866" 01:17:46 INFO [http_cmd]: http status code OK 01:17:46 ERROR [http_cmd]: ERROR 01:17:46 DEBUG --- status code --- 01:17:46 ERROR --- status code --- 01:17:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:17:46 DEBUG --- http response --- 01:17:46 ERROR --- http response --- 01:17:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:17:46 DEBUG ----- output ----- 01:17:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485866" 01:17:46 DEBUG { 01:17:46 DEBUG "status": "success", 01:17:46 DEBUG "data": { 01:17:46 DEBUG "resultType": "vector", 01:17:46 DEBUG "result": [ 01:17:46 DEBUG { 01:17:46 DEBUG "metric": { 01:17:46 DEBUG "instance": "10.142.0.107:9100" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG [...] 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG "instance": "10.142.0.43:9100" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 1713485866, 01:17:46 DEBUG "0.8430333333333323" 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG } 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485866" 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485866" 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485866" 01:17:46 INFO [http_cmd]: http status code OK 01:17:46 DEBUG --- status code --- 01:17:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:46 DEBUG --- http response --- 01:17:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:17:46 INFO [http_cmd]: http status code OK 01:17:46 DEBUG --- status code --- 01:17:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:46 DEBUG --- http response --- 01:17:46 INFO [http_cmd]: http status code OK 01:17:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:46 DEBUG --- status code --- 01:17:46 DEBUG ----- output ----- 01:17:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:46 DEBUG { 01:17:46 DEBUG --- http response --- 01:17:46 DEBUG "status": "success", 01:17:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:17:46 DEBUG "data": { 01:17:46 DEBUG ----- output ----- 01:17:46 DEBUG "resultType": "vector", 01:17:46 DEBUG { 01:17:46 DEBUG "result": [ 01:17:46 DEBUG "status": "success", 01:17:46 DEBUG { 01:17:46 DEBUG "data": { 01:17:46 DEBUG "metric": { 01:17:46 DEBUG "resultType": "vector", 01:17:46 DEBUG "instance": "10.142.0.107:9100" 01:17:46 DEBUG "result": [ 01:17:46 DEBUG }, 01:17:46 DEBUG { 01:17:46 DEBUG "value": [ 01:17:46 DEBUG "metric": { 01:17:46 INFO [http_cmd]: http status code OK 01:17:46 DEBUG 01:17:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:17:46 DEBUG --- status code --- 01:17:46 DEBUG 01:17:46 DEBUG }, 01:17:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:46 DEBUG 01:17:46 DEBUG "value": [ 01:17:46 DEBUG --- http response --- 01:17:46 DEBUG [...] 01:17:46 DEBUG 01:17:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG ----- output ----- 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG { 01:17:46 DEBUG 01:17:46 DEBUG [...] 01:17:46 DEBUG "status": "success", 01:17:46 DEBUG "instance": "10.142.0.43:9100" 01:17:46 DEBUG 01:17:46 DEBUG "data": { 01:17:46 DEBUG }, 01:17:46 DEBUG 01:17:46 DEBUG "resultType": "vector", 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 01:17:46 DEBUG "result": [ 01:17:46 DEBUG 1713485866, 01:17:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:17:46 DEBUG { 01:17:46 DEBUG "350787697664" 01:17:46 DEBUG }, 01:17:46 DEBUG "metric": { 01:17:46 DEBUG ] 01:17:46 DEBUG "value": [ 01:17:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:17:46 DEBUG } 01:17:46 DEBUG 1713485866, 01:17:46 DEBUG }, 01:17:46 DEBUG ] 01:17:46 DEBUG "0" 01:17:46 DEBUG "value": [ 01:17:46 DEBUG } 01:17:46 DEBUG ] 01:17:46 DEBUG 01:17:46 DEBUG } 01:17:46 DEBUG } 01:17:46 DEBUG 01:17:46 DEBUG ] 01:17:46 DEBUG 01:17:46 DEBUG } 01:17:46 DEBUG [...] 01:17:46 DEBUG } 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 1713485866, 01:17:46 DEBUG "1910.3999999999999" 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG } 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485866" 01:17:46 INFO [http_cmd]: http status code OK 01:17:46 DEBUG --- status code --- 01:17:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:46 DEBUG --- http response --- 01:17:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:17:46 DEBUG ----- output ----- 01:17:46 DEBUG { 01:17:46 DEBUG "status": "success", 01:17:46 DEBUG "data": { 01:17:46 DEBUG "resultType": "vector", 01:17:46 DEBUG "result": [ 01:17:46 DEBUG { 01:17:46 DEBUG "metric": { 01:17:46 DEBUG "pod": "ds-idrepo-0" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG [...] 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG "pod": "ds-cts-2" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 1713485866, 01:17:46 DEBUG "0" 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG } 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485866" 01:17:46 INFO [http_cmd]: http status code OK 01:17:46 DEBUG --- status code --- 01:17:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:46 DEBUG --- http response --- 01:17:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:46 DEBUG ----- output ----- 01:17:46 DEBUG { 01:17:46 DEBUG "status": "success", 01:17:46 DEBUG "data": { 01:17:46 DEBUG "resultType": "vector", 01:17:46 DEBUG "result": [ 01:17:46 DEBUG { 01:17:46 DEBUG "metric": { 01:17:46 DEBUG "instance": "10.142.0.107:9100" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG [...] 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG "instance": "10.142.0.43:9100" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 1713485866, 01:17:46 DEBUG "39975920.9" 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG } 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485866" 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485866" 01:17:46 INFO [http_cmd]: http status code OK 01:17:46 DEBUG --- status code --- 01:17:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:46 DEBUG --- http response --- 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485866" 01:17:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:17:46 DEBUG ----- output ----- 01:17:46 DEBUG { 01:17:46 DEBUG "status": "success", 01:17:46 DEBUG "data": { 01:17:46 DEBUG "resultType": "vector", 01:17:46 DEBUG "result": [ 01:17:46 DEBUG { 01:17:46 DEBUG "metric": { 01:17:46 DEBUG "pod": "ds-cts-0" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG [...] 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG "pod": "ds-cts-2" 01:17:46 DEBUG }, 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485866" 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 1713485866, 01:17:46 DEBUG "0" 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG } 01:17:46 INFO [http_cmd]: http status code OK 01:17:46 DEBUG --- status code --- 01:17:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:46 DEBUG --- http response --- 01:17:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:46 DEBUG ----- output ----- 01:17:46 DEBUG { 01:17:46 DEBUG "status": "success", 01:17:46 DEBUG "data": { 01:17:46 DEBUG "resultType": "vector", 01:17:46 DEBUG "result": [ 01:17:46 DEBUG { 01:17:46 DEBUG "metric": { 01:17:46 DEBUG "pod": "ds-idrepo-0" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG [...] 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG "pod": "ds-idrepo-2" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 1713485866, 01:17:46 DEBUG "0" 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG } 01:17:46 INFO [http_cmd]: http status code OK 01:17:46 DEBUG --- status code --- 01:17:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:46 DEBUG --- http response --- 01:17:46 INFO [http_cmd]: http status code OK 01:17:46 DEBUG --- status code --- 01:17:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:46 DEBUG --- http response --- 01:17:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:17:46 DEBUG ----- output ----- 01:17:46 DEBUG { 01:17:46 DEBUG "status": "success", 01:17:46 DEBUG "data": { 01:17:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:46 DEBUG "resultType": "vector", 01:17:46 DEBUG ----- output ----- 01:17:46 DEBUG "result": [ 01:17:46 DEBUG { 01:17:46 DEBUG { 01:17:46 DEBUG "status": "success", 01:17:46 DEBUG "metric": { 01:17:46 DEBUG "data": { 01:17:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:17:46 DEBUG "resultType": "vector", 01:17:46 DEBUG }, 01:17:46 DEBUG "result": [ 01:17:46 DEBUG "value": [ 01:17:46 DEBUG { 01:17:46 DEBUG 01:17:46 DEBUG "metric": { 01:17:46 DEBUG 01:17:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:17:46 DEBUG 01:17:46 DEBUG }, 01:17:46 DEBUG [...] 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG "pod": "idm-b85f9d978-wnhxb" 01:17:46 DEBUG [...] 01:17:46 DEBUG }, 01:17:46 DEBUG 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 01:17:46 DEBUG 1713485866, 01:17:46 DEBUG 01:17:46 DEBUG "0" 01:17:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:17:46 DEBUG ] 01:17:46 DEBUG }, 01:17:46 DEBUG } 01:17:46 DEBUG "value": [ 01:17:46 DEBUG ] 01:17:46 DEBUG 1713485866, 01:17:46 DEBUG } 01:17:46 DEBUG "1079.8" 01:17:46 DEBUG } 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG } 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485866" 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485866" 01:17:46 INFO [http_cmd]: http status code OK 01:17:46 DEBUG --- status code --- 01:17:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:46 DEBUG --- http response --- 01:17:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:46 DEBUG ----- output ----- 01:17:46 DEBUG { 01:17:46 DEBUG "status": "success", 01:17:46 DEBUG "data": { 01:17:46 DEBUG "resultType": "vector", 01:17:46 DEBUG "result": [ 01:17:46 DEBUG { 01:17:46 DEBUG "metric": { 01:17:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG [...] 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 1713485866, 01:17:46 DEBUG "0" 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG } 01:17:46 INFO [http_cmd]: http status code OK 01:17:46 DEBUG --- status code --- 01:17:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:46 DEBUG --- http response --- 01:17:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:46 DEBUG ----- output ----- 01:17:46 DEBUG { 01:17:46 DEBUG "status": "success", 01:17:46 DEBUG "data": { 01:17:46 DEBUG "resultType": "vector", 01:17:46 DEBUG "result": [ 01:17:46 DEBUG { 01:17:46 DEBUG "metric": { 01:17:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG [...] 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 1713485866, 01:17:46 DEBUG "8.028090397618103" 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG } 01:17:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485866" 01:17:46 INFO [http_cmd]: http status code OK 01:17:46 DEBUG --- status code --- 01:17:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:46 DEBUG --- http response --- 01:17:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 01:17:46 DEBUG ----- output ----- 01:17:46 DEBUG { 01:17:46 DEBUG "status": "success", 01:17:46 DEBUG "data": { 01:17:46 DEBUG "resultType": "vector", 01:17:46 DEBUG "result": [ 01:17:46 DEBUG { 01:17:46 DEBUG "metric": { 01:17:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG [...] 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG 01:17:46 DEBUG "pod": "idm-b85f9d978-wnhxb" 01:17:46 DEBUG }, 01:17:46 DEBUG "value": [ 01:17:46 DEBUG 1713485866, 01:17:46 DEBUG "0" 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG ] 01:17:46 DEBUG } 01:17:46 DEBUG } 01:17:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485867" 01:17:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485867" 01:17:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485867" 01:17:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485867" 01:17:47 INFO [http_cmd]: http status code OK 01:17:47 DEBUG --- status code --- 01:17:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:47 DEBUG --- http response --- 01:17:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:17:47 DEBUG ----- output ----- 01:17:47 DEBUG { 01:17:47 DEBUG "status": "success", 01:17:47 DEBUG "data": { 01:17:47 DEBUG "resultType": "vector", 01:17:47 DEBUG "result": [ 01:17:47 DEBUG { 01:17:47 DEBUG "metric": { 01:17:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG [...] 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 1713485867, 01:17:47 DEBUG "0" 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG } 01:17:47 INFO [http_cmd]: http status code OK 01:17:47 DEBUG --- status code --- 01:17:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:47 DEBUG --- http response --- 01:17:47 INFO [http_cmd]: http status code OK 01:17:47 DEBUG --- status code --- 01:17:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:47 DEBUG --- http response --- 01:17:47 DEBUG ----- output ----- 01:17:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:47 DEBUG { 01:17:47 DEBUG ----- output ----- 01:17:47 DEBUG "status": "success", 01:17:47 DEBUG { 01:17:47 DEBUG "data": { 01:17:47 DEBUG "status": "success", 01:17:47 DEBUG "resultType": "vector", 01:17:47 DEBUG "data": { 01:17:47 DEBUG "result": [ 01:17:47 DEBUG "resultType": "vector", 01:17:47 DEBUG { 01:17:47 DEBUG "result": [ 01:17:47 INFO [http_cmd]: http status code OK 01:17:47 DEBUG "metric": { 01:17:47 DEBUG { 01:17:47 DEBUG --- status code --- 01:17:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:17:47 DEBUG "metric": { 01:17:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:47 DEBUG }, 01:17:47 DEBUG "pod": "ds-idrepo-0" 01:17:47 DEBUG --- http response --- 01:17:47 DEBUG "value": [ 01:17:47 DEBUG }, 01:17:47 DEBUG 01:17:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 01:17:47 DEBUG ----- output ----- 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG { 01:17:47 DEBUG 01:17:47 DEBUG [...] 01:17:47 DEBUG "status": "success", 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG "data": { 01:17:47 DEBUG [...] 01:17:47 DEBUG 01:17:47 DEBUG "resultType": "vector", 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG "result": [ 01:17:47 DEBUG 01:17:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:17:47 DEBUG { 01:17:47 DEBUG 01:17:47 DEBUG }, 01:17:47 DEBUG "metric": { 01:17:47 DEBUG "pod": "ds-idrepo-2" 01:17:47 DEBUG "value": [ 01:17:47 DEBUG "pod": "ds-idrepo-0" 01:17:47 DEBUG }, 01:17:47 DEBUG 1713485867, 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG "1080.0333333333333" 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 1713485867, 01:17:47 DEBUG ] 01:17:47 DEBUG 01:17:47 DEBUG "0" 01:17:47 DEBUG } 01:17:47 DEBUG 01:17:47 DEBUG ] 01:17:47 DEBUG ] 01:17:47 DEBUG 01:17:47 DEBUG } 01:17:47 DEBUG } 01:17:47 DEBUG [...] 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG 01:17:47 DEBUG } 01:17:47 DEBUG 01:17:47 DEBUG } 01:17:47 DEBUG 01:17:47 DEBUG "pod": "ds-idrepo-2" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 1713485867, 01:17:47 DEBUG "0" 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG } 01:17:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485867" 01:17:47 INFO [http_cmd]: http status code OK 01:17:47 DEBUG --- status code --- 01:17:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:47 DEBUG --- http response --- 01:17:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:47 DEBUG ----- output ----- 01:17:47 DEBUG { 01:17:47 DEBUG "status": "success", 01:17:47 DEBUG "data": { 01:17:47 DEBUG "resultType": "vector", 01:17:47 DEBUG "result": [ 01:17:47 DEBUG { 01:17:47 DEBUG "metric": { 01:17:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG [...] 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 1713485867, 01:17:47 DEBUG "1079.8" 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG } 01:17:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485867" 01:17:47 INFO [http_cmd]: http status code OK 01:17:47 DEBUG --- status code --- 01:17:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:47 DEBUG --- http response --- 01:17:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:47 DEBUG ----- output ----- 01:17:47 DEBUG { 01:17:47 DEBUG "status": "success", 01:17:47 DEBUG "data": { 01:17:47 DEBUG "resultType": "vector", 01:17:47 DEBUG "result": [ 01:17:47 DEBUG { 01:17:47 DEBUG "metric": { 01:17:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG [...] 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 1713485867, 01:17:47 DEBUG "3232259.5" 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG } 01:17:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485867" 01:17:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485867" 01:17:47 INFO [http_cmd]: http status code OK 01:17:47 DEBUG --- status code --- 01:17:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:47 DEBUG --- http response --- 01:17:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:47 DEBUG ----- output ----- 01:17:47 DEBUG { 01:17:47 DEBUG "status": "success", 01:17:47 DEBUG "data": { 01:17:47 DEBUG "resultType": "vector", 01:17:47 DEBUG "result": [ 01:17:47 DEBUG { 01:17:47 DEBUG "metric": { 01:17:47 DEBUG "instance": "10.142.0.107:9100" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG [...] 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG "instance": "10.142.0.43:9100" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 1713485867, 01:17:47 DEBUG "39981134.96666667" 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG } 01:17:47 INFO [http_cmd]: http status code OK 01:17:47 DEBUG --- status code --- 01:17:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:47 DEBUG --- http response --- 01:17:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:17:47 DEBUG ----- output ----- 01:17:47 DEBUG { 01:17:47 DEBUG "status": "success", 01:17:47 DEBUG "data": { 01:17:47 DEBUG "resultType": "vector", 01:17:47 DEBUG "result": [ 01:17:47 DEBUG { 01:17:47 DEBUG "metric": { 01:17:47 DEBUG "pod": "ds-cts-0" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG [...] 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG "pod": "ds-cts-2" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 1713485867, 01:17:47 DEBUG "0.00016666666666666666" 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG } 01:17:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485867" 01:17:47 INFO [http_cmd]: http status code OK 01:17:47 DEBUG --- status code --- 01:17:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:47 DEBUG --- http response --- 01:17:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:17:47 DEBUG ----- output ----- 01:17:47 DEBUG { 01:17:47 DEBUG "status": "success", 01:17:47 DEBUG "data": { 01:17:47 DEBUG "resultType": "vector", 01:17:47 DEBUG "result": [ 01:17:47 DEBUG { 01:17:47 DEBUG "metric": { 01:17:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG [...] 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 1713485867, 01:17:47 DEBUG "20.076" 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG } 01:17:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485867" 01:17:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485867" 01:17:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485867" 01:17:47 INFO [http_cmd]: http status code OK 01:17:47 DEBUG --- status code --- 01:17:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:47 DEBUG --- http response --- 01:17:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:17:47 DEBUG ----- output ----- 01:17:47 DEBUG { 01:17:47 DEBUG "status": "success", 01:17:47 DEBUG "data": { 01:17:47 DEBUG "resultType": "vector", 01:17:47 DEBUG "result": [ 01:17:47 DEBUG { 01:17:47 DEBUG "metric": { 01:17:47 DEBUG "pod": "ds-cts-0" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG [...] 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG "pod": "ds-cts-2" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 1713485867, 01:17:47 DEBUG "607.3333333333334" 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG } 01:17:47 INFO [http_cmd]: http status code OK 01:17:47 DEBUG --- status code --- 01:17:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:47 DEBUG --- http response --- 01:17:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:47 DEBUG ----- output ----- 01:17:47 DEBUG { 01:17:47 DEBUG "status": "success", 01:17:47 DEBUG "data": { 01:17:47 DEBUG "resultType": "vector", 01:17:47 DEBUG "result": [ 01:17:47 DEBUG { 01:17:47 DEBUG "metric": { 01:17:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG [...] 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 1713485867, 01:17:47 DEBUG "7547334656" 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG } 01:17:47 INFO [http_cmd]: http status code OK 01:17:47 DEBUG --- status code --- 01:17:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:17:47 DEBUG --- http response --- 01:17:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:17:47 DEBUG ----- output ----- 01:17:47 DEBUG { 01:17:47 DEBUG "status": "success", 01:17:47 DEBUG "data": { 01:17:47 DEBUG "resultType": "vector", 01:17:47 DEBUG "result": [ 01:17:47 DEBUG { 01:17:47 DEBUG "metric": { 01:17:47 DEBUG "instance": "10.142.0.107:9100" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG [...] 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG 01:17:47 DEBUG "instance": "10.142.0.43:9100" 01:17:47 DEBUG }, 01:17:47 DEBUG "value": [ 01:17:47 DEBUG 1713485867, 01:17:47 DEBUG "0.8786666666666668" 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG ] 01:17:47 DEBUG } 01:17:47 DEBUG } 01:17:48 INFO 01:17:48 INFO [loop_until]: kubectl --namespace=xlou top pods 01:17:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:17:48 INFO [loop_until]: OK (rc = 0) 01:17:48 DEBUG --- stdout --- 01:17:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 8413m 9446Mi am-6fd554f8f7-hvvkw 6657m 8902Mi am-6fd554f8f7-pmj4b 7500m 9077Mi ds-cts-0 15325m 12564Mi ds-cts-1 11915m 12627Mi ds-cts-2 12789m 12655Mi ds-idrepo-0 3235m 3248Mi ds-idrepo-1 3218m 3167Mi ds-idrepo-2 3143m 3040Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 4m 1215Mi lodemon-5d46798758-5h2st 11m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7782m 7102Mi 01:17:48 DEBUG --- stderr --- 01:17:48 DEBUG 01:17:48 INFO 01:17:48 INFO [loop_until]: kubectl --namespace=xlou top node 01:17:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:17:48 INFO [loop_until]: OK (rc = 0) 01:17:48 DEBUG --- stdout --- 01:17:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5929Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3278m 10% 6655Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6751m 22% 11636Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3232Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 14968m 50% 15048Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 136m 0% 5230Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8560m 28% 12244Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 15604m 52% 3914Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 7842m 26% 11298Mi 10% gke-xlou-cdm-ds-0f601751-j488 3464m 11% 6093Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 12271m 41% 16683Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3482m 11% 6800Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 13080m 43% 14550Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 8222m 51% 10332Mi 17% 01:17:48 DEBUG --- stderr --- 01:17:48 DEBUG 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485926" 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485926" 01:18:46 ERROR [http_cmd]: ERROR 01:18:46 INFO [http_cmd]: http status code OK 01:18:46 DEBUG --- status code --- 01:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:46 ERROR --- status code --- 01:18:46 DEBUG --- http response --- 01:18:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:18:46 ERROR --- http response --- 01:18:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:46 DEBUG ----- output ----- 01:18:46 DEBUG { 01:18:46 DEBUG "status": "success", 01:18:46 DEBUG "data": { 01:18:46 DEBUG "resultType": "vector", 01:18:46 DEBUG "result": [ 01:18:46 DEBUG { 01:18:46 DEBUG "metric": { 01:18:46 DEBUG "instance": "10.142.0.107:9100" 01:18:46 DEBUG }, 01:18:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:18:46 DEBUG "value": [ 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485926" 01:18:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG [...] 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG "instance": "10.142.0.43:9100" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 1713485926, 01:18:46 DEBUG "0.9034666666666558" 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485926" 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485926" 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485926" 01:18:46 INFO [http_cmd]: http status code OK 01:18:46 DEBUG --- status code --- 01:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:46 DEBUG --- http response --- 01:18:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:18:46 INFO [http_cmd]: http status code OK 01:18:46 DEBUG --- status code --- 01:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:46 DEBUG --- http response --- 01:18:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:46 DEBUG ----- output ----- 01:18:46 DEBUG { 01:18:46 DEBUG "status": "success", 01:18:46 DEBUG "data": { 01:18:46 DEBUG "resultType": "vector", 01:18:46 DEBUG "result": [ 01:18:46 DEBUG { 01:18:46 DEBUG "metric": { 01:18:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG [...] 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 1713485926, 01:18:46 DEBUG "1921.6666666666667" 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:46 INFO [http_cmd]: http status code OK 01:18:46 DEBUG --- status code --- 01:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:46 DEBUG --- http response --- 01:18:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:46 DEBUG ----- output ----- 01:18:46 DEBUG { 01:18:46 DEBUG "status": "success", 01:18:46 DEBUG "data": { 01:18:46 DEBUG "resultType": "vector", 01:18:46 DEBUG "result": [ 01:18:46 DEBUG { 01:18:46 DEBUG "metric": { 01:18:46 DEBUG "instance": "10.142.0.107:9100" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG [...] 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG "instance": "10.142.0.43:9100" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 1713485926, 01:18:46 DEBUG "352632679424" 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:46 INFO [http_cmd]: http status code OK 01:18:46 DEBUG --- status code --- 01:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:46 DEBUG --- http response --- 01:18:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:18:46 DEBUG ----- output ----- 01:18:46 DEBUG { 01:18:46 DEBUG "status": "success", 01:18:46 DEBUG "data": { 01:18:46 DEBUG "resultType": "vector", 01:18:46 DEBUG "result": [ 01:18:46 DEBUG { 01:18:46 DEBUG "metric": { 01:18:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG [...] 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 1713485926, 01:18:46 DEBUG "0" 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485926" 01:18:46 INFO [http_cmd]: http status code OK 01:18:46 DEBUG --- status code --- 01:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:46 DEBUG --- http response --- 01:18:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:18:46 DEBUG ----- output ----- 01:18:46 DEBUG { 01:18:46 DEBUG "status": "success", 01:18:46 DEBUG "data": { 01:18:46 DEBUG "resultType": "vector", 01:18:46 DEBUG "result": [ 01:18:46 DEBUG { 01:18:46 DEBUG "metric": { 01:18:46 DEBUG "pod": "ds-idrepo-0" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG [...] 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG "pod": "ds-cts-2" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 1713485926, 01:18:46 DEBUG "0" 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485926" 01:18:46 INFO [http_cmd]: http status code OK 01:18:46 DEBUG --- status code --- 01:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:46 DEBUG --- http response --- 01:18:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:46 DEBUG ----- output ----- 01:18:46 DEBUG { 01:18:46 DEBUG "status": "success", 01:18:46 DEBUG "data": { 01:18:46 DEBUG "resultType": "vector", 01:18:46 DEBUG "result": [ 01:18:46 DEBUG { 01:18:46 DEBUG "metric": { 01:18:46 DEBUG "instance": "10.142.0.107:9100" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG [...] 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG "instance": "10.142.0.43:9100" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 1713485926, 01:18:46 DEBUG "37786216.56666666" 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485926" 01:18:46 INFO [http_cmd]: http status code OK 01:18:46 DEBUG --- status code --- 01:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:46 DEBUG --- http response --- 01:18:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:46 DEBUG ----- output ----- 01:18:46 DEBUG { 01:18:46 DEBUG "status": "success", 01:18:46 DEBUG "data": { 01:18:46 DEBUG "resultType": "vector", 01:18:46 DEBUG "result": [ 01:18:46 DEBUG { 01:18:46 DEBUG "metric": { 01:18:46 DEBUG "pod": "ds-idrepo-0" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG [...] 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG "pod": "ds-idrepo-2" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 1713485926, 01:18:46 DEBUG "0" 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485926" 01:18:46 INFO [http_cmd]: http status code OK 01:18:46 DEBUG --- status code --- 01:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:46 DEBUG --- http response --- 01:18:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:18:46 DEBUG ----- output ----- 01:18:46 DEBUG { 01:18:46 DEBUG "status": "success", 01:18:46 DEBUG "data": { 01:18:46 DEBUG "resultType": "vector", 01:18:46 DEBUG "result": [ 01:18:46 DEBUG { 01:18:46 DEBUG "metric": { 01:18:46 DEBUG "pod": "ds-cts-0" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG [...] 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG "pod": "ds-cts-2" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 1713485926, 01:18:46 DEBUG "0" 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485926" 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485926" 01:18:46 INFO [http_cmd]: http status code OK 01:18:46 INFO [http_cmd]: http status code OK 01:18:46 DEBUG --- status code --- 01:18:46 DEBUG --- status code --- 01:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:46 DEBUG --- http response --- 01:18:46 DEBUG --- http response --- 01:18:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:46 DEBUG ----- output ----- 01:18:46 DEBUG ----- output ----- 01:18:46 DEBUG { 01:18:46 DEBUG { 01:18:46 DEBUG "status": "success", 01:18:46 DEBUG "status": "success", 01:18:46 DEBUG "data": { 01:18:46 DEBUG "data": { 01:18:46 DEBUG "resultType": "vector", 01:18:46 DEBUG "resultType": "vector", 01:18:46 DEBUG "result": [ 01:18:46 DEBUG "result": [ 01:18:46 DEBUG { 01:18:46 DEBUG { 01:18:46 DEBUG "metric": { 01:18:46 DEBUG "metric": { 01:18:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:18:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:18:46 DEBUG }, 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG [...] 01:18:46 DEBUG [...] 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:18:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:18:46 DEBUG }, 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 1713485926, 01:18:46 DEBUG 1713485926, 01:18:46 DEBUG "1018.8666666666667" 01:18:46 DEBUG "0" 01:18:46 DEBUG ] 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:46 DEBUG ] 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485926" 01:18:46 INFO [http_cmd]: http status code OK 01:18:46 DEBUG --- status code --- 01:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:46 DEBUG --- http response --- 01:18:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:46 DEBUG ----- output ----- 01:18:46 DEBUG { 01:18:46 DEBUG "status": "success", 01:18:46 DEBUG "data": { 01:18:46 DEBUG "resultType": "vector", 01:18:46 DEBUG "result": [ 01:18:46 DEBUG { 01:18:46 DEBUG "metric": { 01:18:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG [...] 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 1713485926, 01:18:46 DEBUG "8.040687679814402" 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485926" 01:18:46 INFO [http_cmd]: http status code OK 01:18:46 DEBUG --- status code --- 01:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:46 DEBUG --- http response --- 01:18:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:46 DEBUG ----- output ----- 01:18:46 DEBUG { 01:18:46 DEBUG "status": "success", 01:18:46 DEBUG "data": { 01:18:46 DEBUG "resultType": "vector", 01:18:46 DEBUG "result": [ 01:18:46 DEBUG { 01:18:46 DEBUG "metric": { 01:18:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG [...] 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 1713485926, 01:18:46 DEBUG "0" 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485926" 01:18:46 INFO [http_cmd]: http status code OK 01:18:46 DEBUG --- status code --- 01:18:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:46 DEBUG --- http response --- 01:18:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:46 DEBUG ----- output ----- 01:18:46 DEBUG { 01:18:46 DEBUG "status": "success", 01:18:46 DEBUG "data": { 01:18:46 DEBUG "resultType": "vector", 01:18:46 DEBUG "result": [ 01:18:46 DEBUG { 01:18:46 DEBUG "metric": { 01:18:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG [...] 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG 01:18:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:18:46 DEBUG }, 01:18:46 DEBUG "value": [ 01:18:46 DEBUG 1713485926, 01:18:46 DEBUG "0" 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG ] 01:18:46 DEBUG } 01:18:46 DEBUG } 01:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485927" 01:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485927" 01:18:47 INFO [http_cmd]: http status code OK 01:18:47 DEBUG --- status code --- 01:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:47 DEBUG --- http response --- 01:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:18:47 DEBUG ----- output ----- 01:18:47 DEBUG { 01:18:47 DEBUG "status": "success", 01:18:47 DEBUG "data": { 01:18:47 DEBUG "resultType": "vector", 01:18:47 DEBUG "result": [ 01:18:47 DEBUG { 01:18:47 DEBUG "metric": { 01:18:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG [...] 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 1713485927, 01:18:47 DEBUG "0" 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG } 01:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485927" 01:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485927" 01:18:47 INFO [http_cmd]: http status code OK 01:18:47 DEBUG --- status code --- 01:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:47 DEBUG --- http response --- 01:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:47 DEBUG ----- output ----- 01:18:47 DEBUG { 01:18:47 DEBUG "status": "success", 01:18:47 DEBUG "data": { 01:18:47 DEBUG "resultType": "vector", 01:18:47 DEBUG "result": [ 01:18:47 DEBUG { 01:18:47 DEBUG "metric": { 01:18:47 DEBUG "pod": "ds-idrepo-0" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG [...] 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG "pod": "ds-idrepo-2" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 1713485927, 01:18:47 DEBUG "0" 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG } 01:18:47 INFO [http_cmd]: http status code OK 01:18:47 DEBUG --- status code --- 01:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:47 DEBUG --- http response --- 01:18:47 INFO [http_cmd]: http status code OK 01:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:47 DEBUG --- status code --- 01:18:47 DEBUG ----- output ----- 01:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:47 DEBUG { 01:18:47 DEBUG --- http response --- 01:18:47 DEBUG "status": "success", 01:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:47 DEBUG "data": { 01:18:47 DEBUG ----- output ----- 01:18:47 DEBUG "resultType": "vector", 01:18:47 DEBUG { 01:18:47 DEBUG "result": [ 01:18:47 DEBUG { 01:18:47 DEBUG "metric": { 01:18:47 DEBUG "pod": "ds-idrepo-0" 01:18:47 DEBUG "status": "success", 01:18:47 DEBUG }, 01:18:47 DEBUG "data": { 01:18:47 DEBUG "value": [ 01:18:47 DEBUG "resultType": "vector", 01:18:47 DEBUG 01:18:47 DEBUG "result": [ 01:18:47 DEBUG 01:18:47 DEBUG { 01:18:47 DEBUG 01:18:47 DEBUG "metric": { 01:18:47 DEBUG [...] 01:18:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:18:47 DEBUG 01:18:47 DEBUG }, 01:18:47 DEBUG 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG "pod": "ds-idrepo-2" 01:18:47 DEBUG 01:18:47 DEBUG }, 01:18:47 DEBUG 01:18:47 DEBUG "value": [ 01:18:47 DEBUG [...] 01:18:47 DEBUG 1713485927, 01:18:47 DEBUG 01:18:47 DEBUG "0" 01:18:47 DEBUG 01:18:47 DEBUG ] 01:18:47 DEBUG 01:18:47 DEBUG } 01:18:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:18:47 DEBUG ] 01:18:47 DEBUG }, 01:18:47 DEBUG } 01:18:47 DEBUG "value": [ 01:18:47 DEBUG } 01:18:47 DEBUG 1713485927, 01:18:47 DEBUG "1021.1" 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG } 01:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485927" 01:18:47 INFO [http_cmd]: http status code OK 01:18:47 DEBUG --- status code --- 01:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:47 DEBUG --- http response --- 01:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:47 DEBUG ----- output ----- 01:18:47 DEBUG { 01:18:47 DEBUG "status": "success", 01:18:47 DEBUG "data": { 01:18:47 DEBUG "resultType": "vector", 01:18:47 DEBUG "result": [ 01:18:47 DEBUG { 01:18:47 DEBUG "metric": { 01:18:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG [...] 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 1713485927, 01:18:47 DEBUG "1020.1" 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG } 01:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485927" 01:18:47 INFO [http_cmd]: http status code OK 01:18:47 DEBUG --- status code --- 01:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:47 DEBUG --- http response --- 01:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:47 DEBUG ----- output ----- 01:18:47 DEBUG { 01:18:47 DEBUG "status": "success", 01:18:47 DEBUG "data": { 01:18:47 DEBUG "resultType": "vector", 01:18:47 DEBUG "result": [ 01:18:47 DEBUG { 01:18:47 DEBUG "metric": { 01:18:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG [...] 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 1713485927, 01:18:47 DEBUG "3181343" 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG } 01:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485927" 01:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485927" 01:18:47 INFO [http_cmd]: http status code OK 01:18:47 DEBUG --- status code --- 01:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:47 INFO [http_cmd]: http status code OK 01:18:47 DEBUG --- http response --- 01:18:47 DEBUG --- status code --- 01:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:47 DEBUG ----- output ----- 01:18:47 DEBUG --- http response --- 01:18:47 DEBUG { 01:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:18:47 DEBUG "status": "success", 01:18:47 DEBUG ----- output ----- 01:18:47 DEBUG "data": { 01:18:47 DEBUG { 01:18:47 DEBUG "resultType": "vector", 01:18:47 DEBUG "status": "success", 01:18:47 DEBUG "result": [ 01:18:47 DEBUG "data": { 01:18:47 DEBUG { 01:18:47 DEBUG "resultType": "vector", 01:18:47 DEBUG "metric": { 01:18:47 DEBUG "result": [ 01:18:47 DEBUG "instance": "10.142.0.107:9100" 01:18:47 DEBUG { 01:18:47 DEBUG }, 01:18:47 DEBUG "metric": { 01:18:47 DEBUG "value": [ 01:18:47 DEBUG "pod": "ds-cts-0" 01:18:47 DEBUG 01:18:47 DEBUG }, 01:18:47 DEBUG 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG [...] 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG [...] 01:18:47 DEBUG "instance": "10.142.0.43:9100" 01:18:47 DEBUG 01:18:47 DEBUG }, 01:18:47 DEBUG 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 01:18:47 DEBUG 1713485927, 01:18:47 DEBUG "pod": "ds-cts-2" 01:18:47 DEBUG "37787627.96666667" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485927" 01:18:47 DEBUG 1713485927, 01:18:47 DEBUG ] 01:18:47 DEBUG "0.00030000000000000003" 01:18:47 DEBUG } 01:18:47 DEBUG ] 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG } 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG } 01:18:47 DEBUG } 01:18:47 INFO [http_cmd]: http status code OK 01:18:47 DEBUG --- status code --- 01:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:47 DEBUG --- http response --- 01:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:18:47 DEBUG ----- output ----- 01:18:47 DEBUG { 01:18:47 DEBUG "status": "success", 01:18:47 DEBUG "data": { 01:18:47 DEBUG "resultType": "vector", 01:18:47 DEBUG "result": [ 01:18:47 DEBUG { 01:18:47 DEBUG "metric": { 01:18:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG [...] 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 1713485927, 01:18:47 DEBUG "20.076" 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG } 01:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485927" 01:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485927" 01:18:47 INFO [http_cmd]: http status code OK 01:18:47 DEBUG --- status code --- 01:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:47 DEBUG --- http response --- 01:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:47 DEBUG ----- output ----- 01:18:47 DEBUG { 01:18:47 DEBUG "status": "success", 01:18:47 DEBUG "data": { 01:18:47 DEBUG "resultType": "vector", 01:18:47 DEBUG "result": [ 01:18:47 DEBUG { 01:18:47 DEBUG "metric": { 01:18:47 DEBUG "instance": "10.142.0.107:9100" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485927" 01:18:47 DEBUG 01:18:47 DEBUG [...] 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG "instance": "10.142.0.43:9100" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 1713485927, 01:18:47 DEBUG "1.4320000000000017" 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG } 01:18:47 INFO [http_cmd]: http status code OK 01:18:47 DEBUG --- status code --- 01:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:47 DEBUG --- http response --- 01:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:18:47 DEBUG ----- output ----- 01:18:47 DEBUG { 01:18:47 DEBUG "status": "success", 01:18:47 DEBUG "data": { 01:18:47 DEBUG "resultType": "vector", 01:18:47 DEBUG "result": [ 01:18:47 DEBUG { 01:18:47 DEBUG "metric": { 01:18:47 DEBUG "pod": "ds-cts-0" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG [...] 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG "pod": "ds-cts-2" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 1713485927, 01:18:47 DEBUG "755.4" 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG } 01:18:47 INFO [http_cmd]: http status code OK 01:18:47 DEBUG --- status code --- 01:18:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:18:47 DEBUG --- http response --- 01:18:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:18:47 DEBUG ----- output ----- 01:18:47 DEBUG { 01:18:47 DEBUG "status": "success", 01:18:47 DEBUG "data": { 01:18:47 DEBUG "resultType": "vector", 01:18:47 DEBUG "result": [ 01:18:47 DEBUG { 01:18:47 DEBUG "metric": { 01:18:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG [...] 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG 01:18:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:18:47 DEBUG }, 01:18:47 DEBUG "value": [ 01:18:47 DEBUG 1713485927, 01:18:47 DEBUG "8309833728" 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG ] 01:18:47 DEBUG } 01:18:47 DEBUG } 01:18:48 INFO 01:18:48 INFO [loop_until]: kubectl --namespace=xlou top pods 01:18:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:18:48 INFO [loop_until]: OK (rc = 0) 01:18:48 DEBUG --- stdout --- 01:18:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 8325m 9450Mi am-6fd554f8f7-hvvkw 6661m 8904Mi am-6fd554f8f7-pmj4b 7261m 9081Mi ds-cts-0 15125m 12498Mi ds-cts-1 14055m 12351Mi ds-cts-2 13484m 12654Mi ds-idrepo-0 3166m 3249Mi ds-idrepo-1 3234m 3168Mi ds-idrepo-2 3107m 3039Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1171Mi idm-b85f9d978-wnhxb 4m 1215Mi lodemon-5d46798758-5h2st 2m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8150m 7830Mi 01:18:48 DEBUG --- stderr --- 01:18:48 DEBUG 01:18:48 INFO 01:18:48 INFO [loop_until]: kubectl --namespace=xlou top node 01:18:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:18:48 INFO [loop_until]: OK (rc = 0) 01:18:48 DEBUG --- stdout --- 01:18:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5932Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3219m 10% 6667Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6555m 21% 11634Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3242Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 14064m 47% 14958Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 5217Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8751m 29% 12242Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 14690m 49% 3910Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 7715m 25% 11308Mi 10% gke-xlou-cdm-ds-0f601751-j488 3319m 11% 6111Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 13742m 46% 16384Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3443m 11% 6809Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 12770m 42% 14262Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 8279m 52% 10936Mi 18% 01:18:48 DEBUG --- stderr --- 01:18:48 DEBUG 01:19:36 WARNING File /tmp/lodemon_data-20240419_004139/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [19/Apr/2024 01:19:36] "GET /monitoring/average?start_time=20240419_001556&stop_time=20240419_001735 HTTP/1.1" 200 - 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713485986" 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485986" 01:19:46 ERROR [http_cmd]: ERROR 01:19:46 ERROR --- status code --- 01:19:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:19:46 ERROR --- http response --- 01:19:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:19:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:19:46 INFO [http_cmd]: http status code OK 01:19:46 DEBUG --- status code --- 01:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:46 DEBUG --- http response --- 01:19:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:46 DEBUG ----- output ----- 01:19:46 DEBUG { 01:19:46 DEBUG "status": "success", 01:19:46 DEBUG "data": { 01:19:46 DEBUG "resultType": "vector", 01:19:46 DEBUG "result": [ 01:19:46 DEBUG { 01:19:46 DEBUG "metric": { 01:19:46 DEBUG "instance": "10.142.0.107:9100" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG [...] 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG "instance": "10.142.0.43:9100" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 1713485986, 01:19:46 DEBUG "0.7749333333333293" 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG } 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485986" 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485986" 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485986" 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485986" 01:19:46 INFO [http_cmd]: http status code OK 01:19:46 DEBUG --- status code --- 01:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:46 DEBUG --- http response --- 01:19:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:19:46 INFO [http_cmd]: http status code OK 01:19:46 DEBUG --- status code --- 01:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:46 DEBUG --- http response --- 01:19:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:46 INFO [http_cmd]: http status code OK 01:19:46 DEBUG ----- output ----- 01:19:46 DEBUG --- status code --- 01:19:46 DEBUG { 01:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:46 DEBUG "status": "success", 01:19:46 DEBUG --- http response --- 01:19:46 DEBUG "data": { 01:19:46 DEBUG "resultType": "vector", 01:19:46 DEBUG "result": [ 01:19:46 DEBUG { 01:19:46 DEBUG "metric": { 01:19:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 01:19:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:46 DEBUG 01:19:46 DEBUG ----- output ----- 01:19:46 DEBUG 01:19:46 DEBUG { 01:19:46 DEBUG [...] 01:19:46 DEBUG "status": "success", 01:19:46 DEBUG 01:19:46 DEBUG "data": { 01:19:46 DEBUG 01:19:46 DEBUG "resultType": "vector", 01:19:46 DEBUG "result": [ 01:19:46 DEBUG 01:19:46 DEBUG { 01:19:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:19:46 DEBUG "metric": { 01:19:46 DEBUG }, 01:19:46 DEBUG "instance": "10.142.0.107:9100" 01:19:46 DEBUG "value": [ 01:19:46 DEBUG }, 01:19:46 DEBUG 1713485986, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG "1929.7" 01:19:46 DEBUG 01:19:46 DEBUG ] 01:19:46 DEBUG 01:19:46 DEBUG } 01:19:46 DEBUG 01:19:46 DEBUG ] 01:19:46 DEBUG [...] 01:19:46 DEBUG } 01:19:46 DEBUG 01:19:46 DEBUG } 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG "instance": "10.142.0.43:9100" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 1713485986, 01:19:46 DEBUG "354082911232" 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG } 01:19:46 INFO [http_cmd]: http status code OK 01:19:46 DEBUG --- status code --- 01:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:46 DEBUG --- http response --- 01:19:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:46 DEBUG ----- output ----- 01:19:46 DEBUG { 01:19:46 DEBUG "status": "success", 01:19:46 DEBUG "data": { 01:19:46 DEBUG "resultType": "vector", 01:19:46 DEBUG "result": [ 01:19:46 DEBUG { 01:19:46 DEBUG "metric": { 01:19:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG [...] 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 1713485986, 01:19:46 DEBUG "961.9822094354578" 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG } 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713485986" 01:19:46 INFO [http_cmd]: http status code OK 01:19:46 DEBUG --- status code --- 01:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:46 DEBUG --- http response --- 01:19:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:19:46 DEBUG ----- output ----- 01:19:46 DEBUG { 01:19:46 DEBUG "status": "success", 01:19:46 DEBUG "data": { 01:19:46 DEBUG "resultType": "vector", 01:19:46 DEBUG "result": [ 01:19:46 DEBUG { 01:19:46 DEBUG "metric": { 01:19:46 DEBUG "pod": "ds-idrepo-0" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG [...] 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG "pod": "ds-cts-2" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 1713485986, 01:19:46 DEBUG "0" 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG } 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485986" 01:19:46 INFO [http_cmd]: http status code OK 01:19:46 DEBUG --- status code --- 01:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:46 DEBUG --- http response --- 01:19:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:46 DEBUG ----- output ----- 01:19:46 DEBUG { 01:19:46 DEBUG "status": "success", 01:19:46 DEBUG "data": { 01:19:46 DEBUG "resultType": "vector", 01:19:46 DEBUG "result": [ 01:19:46 DEBUG { 01:19:46 DEBUG "metric": { 01:19:46 DEBUG "instance": "10.142.0.107:9100" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG [...] 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG "instance": "10.142.0.43:9100" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 1713485986, 01:19:46 DEBUG "13273418" 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG } 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485986" 01:19:46 INFO [http_cmd]: http status code OK 01:19:46 DEBUG --- status code --- 01:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:46 DEBUG --- http response --- 01:19:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485986" 01:19:46 DEBUG ----- output ----- 01:19:46 DEBUG { 01:19:46 DEBUG "status": "success", 01:19:46 DEBUG "data": { 01:19:46 DEBUG "resultType": "vector", 01:19:46 DEBUG "result": [ 01:19:46 DEBUG { 01:19:46 DEBUG "metric": { 01:19:46 DEBUG "pod": "ds-idrepo-0" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG [...] 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG "pod": "ds-idrepo-2" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 1713485986, 01:19:46 DEBUG "0" 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG } 01:19:46 INFO [http_cmd]: http status code OK 01:19:46 DEBUG --- status code --- 01:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:46 DEBUG --- http response --- 01:19:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:19:46 DEBUG ----- output ----- 01:19:46 DEBUG { 01:19:46 DEBUG "status": "success", 01:19:46 DEBUG "data": { 01:19:46 DEBUG "resultType": "vector", 01:19:46 DEBUG "result": [ 01:19:46 DEBUG { 01:19:46 DEBUG "metric": { 01:19:46 DEBUG "pod": "ds-cts-0" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG [...] 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG "pod": "ds-cts-2" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 1713485986, 01:19:46 DEBUG "0" 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG } 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485986" 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485986" 01:19:46 INFO [http_cmd]: http status code OK 01:19:46 DEBUG --- status code --- 01:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:46 DEBUG --- http response --- 01:19:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:46 DEBUG ----- output ----- 01:19:46 DEBUG { 01:19:46 DEBUG "status": "success", 01:19:46 DEBUG "data": { 01:19:46 DEBUG "resultType": "vector", 01:19:46 DEBUG "result": [ 01:19:46 DEBUG { 01:19:46 DEBUG "metric": { 01:19:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG [...] 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 1713485986, 01:19:46 DEBUG "13.333333333333334" 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG } 01:19:46 INFO [http_cmd]: http status code OK 01:19:46 DEBUG --- status code --- 01:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:46 DEBUG --- http response --- 01:19:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:46 DEBUG ----- output ----- 01:19:46 DEBUG { 01:19:46 DEBUG "status": "success", 01:19:46 DEBUG "data": { 01:19:46 DEBUG "resultType": "vector", 01:19:46 DEBUG "result": [ 01:19:46 DEBUG { 01:19:46 DEBUG "metric": { 01:19:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG [...] 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 1713485986, 01:19:46 DEBUG "279652917.9461586" 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG } 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713485986" 01:19:46 INFO [http_cmd]: http status code OK 01:19:46 DEBUG --- status code --- 01:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:46 DEBUG --- http response --- 01:19:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:46 DEBUG ----- output ----- 01:19:46 DEBUG { 01:19:46 DEBUG "status": "success", 01:19:46 DEBUG "data": { 01:19:46 DEBUG "resultType": "vector", 01:19:46 DEBUG "result": [ 01:19:46 DEBUG { 01:19:46 DEBUG "metric": { 01:19:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG [...] 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 1713485986, 01:19:46 DEBUG "4.569950238816045" 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG } 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485986" 01:19:46 INFO [http_cmd]: http status code OK 01:19:46 DEBUG --- status code --- 01:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:46 DEBUG --- http response --- 01:19:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:46 DEBUG ----- output ----- 01:19:46 DEBUG { 01:19:46 DEBUG "status": "success", 01:19:46 DEBUG "data": { 01:19:46 DEBUG "resultType": "vector", 01:19:46 DEBUG "result": [ 01:19:46 DEBUG { 01:19:46 DEBUG "metric": { 01:19:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG [...] 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 1713485986, 01:19:46 DEBUG "0" 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG } 01:19:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485986" 01:19:46 INFO [http_cmd]: http status code OK 01:19:46 DEBUG --- status code --- 01:19:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:46 DEBUG --- http response --- 01:19:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:46 DEBUG ----- output ----- 01:19:46 DEBUG { 01:19:46 DEBUG "status": "success", 01:19:46 DEBUG "data": { 01:19:46 DEBUG "resultType": "vector", 01:19:46 DEBUG "result": [ 01:19:46 DEBUG { 01:19:46 DEBUG "metric": { 01:19:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG [...] 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG 01:19:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:19:46 DEBUG }, 01:19:46 DEBUG "value": [ 01:19:46 DEBUG 1713485986, 01:19:46 DEBUG "4.785161377447709" 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG ] 01:19:46 DEBUG } 01:19:46 DEBUG } 01:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485987" 01:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485987" 01:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485987" 01:19:47 INFO [http_cmd]: http status code OK 01:19:47 DEBUG --- status code --- 01:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:47 DEBUG --- http response --- 01:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:47 DEBUG ----- output ----- 01:19:47 DEBUG { 01:19:47 DEBUG "status": "success", 01:19:47 DEBUG "data": { 01:19:47 DEBUG "resultType": "vector", 01:19:47 DEBUG "result": [ 01:19:47 DEBUG { 01:19:47 DEBUG "metric": { 01:19:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG [...] 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 1713485987, 01:19:47 DEBUG "0" 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG } 01:19:47 INFO [http_cmd]: http status code OK 01:19:47 DEBUG --- status code --- 01:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:47 DEBUG --- http response --- 01:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:47 DEBUG ----- output ----- 01:19:47 DEBUG { 01:19:47 DEBUG "status": "success", 01:19:47 DEBUG "data": { 01:19:47 DEBUG "resultType": "vector", 01:19:47 DEBUG "result": [ 01:19:47 DEBUG { 01:19:47 DEBUG "metric": { 01:19:47 DEBUG "pod": "ds-idrepo-0" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG [...] 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG "pod": "ds-idrepo-2" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 1713485987, 01:19:47 DEBUG "0" 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG } 01:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485987" 01:19:47 INFO [http_cmd]: http status code OK 01:19:47 DEBUG --- status code --- 01:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:47 DEBUG --- http response --- 01:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:47 DEBUG ----- output ----- 01:19:47 DEBUG { 01:19:47 DEBUG "status": "success", 01:19:47 DEBUG "data": { 01:19:47 DEBUG "resultType": "vector", 01:19:47 DEBUG "result": [ 01:19:47 DEBUG { 01:19:47 DEBUG "metric": { 01:19:47 DEBUG "pod": "ds-idrepo-0" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG [...] 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG "pod": "ds-idrepo-2" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 1713485987, 01:19:47 DEBUG "0" 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG } 01:19:47 INFO [http_cmd]: http status code OK 01:19:47 DEBUG --- status code --- 01:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:47 DEBUG --- http response --- 01:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:47 DEBUG ----- output ----- 01:19:47 DEBUG { 01:19:47 DEBUG "status": "success", 01:19:47 DEBUG "data": { 01:19:47 DEBUG "resultType": "vector", 01:19:47 DEBUG "result": [ 01:19:47 DEBUG { 01:19:47 DEBUG "metric": { 01:19:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG [...] 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 1713485987, 01:19:47 DEBUG "13.666666666666666" 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG } 01:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485987" 01:19:47 INFO [http_cmd]: http status code OK 01:19:47 DEBUG --- status code --- 01:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:47 DEBUG --- http response --- 01:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:47 DEBUG ----- output ----- 01:19:47 DEBUG { 01:19:47 DEBUG "status": "success", 01:19:47 DEBUG "data": { 01:19:47 DEBUG "resultType": "vector", 01:19:47 DEBUG "result": [ 01:19:47 DEBUG { 01:19:47 DEBUG "metric": { 01:19:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG [...] 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 1713485987, 01:19:47 DEBUG "13.433333333333334" 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG } 01:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485987" 01:19:47 INFO [http_cmd]: http status code OK 01:19:47 DEBUG --- status code --- 01:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:47 DEBUG --- http response --- 01:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:47 DEBUG ----- output ----- 01:19:47 DEBUG { 01:19:47 DEBUG "status": "success", 01:19:47 DEBUG "data": { 01:19:47 DEBUG "resultType": "vector", 01:19:47 DEBUG "result": [ 01:19:47 DEBUG { 01:19:47 DEBUG "metric": { 01:19:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG [...] 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 1713485987, 01:19:47 DEBUG "3112591" 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG } 01:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485987" 01:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713485987" 01:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485987" 01:19:47 INFO [http_cmd]: http status code OK 01:19:47 DEBUG --- status code --- 01:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:47 DEBUG --- http response --- 01:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:19:47 DEBUG ----- output ----- 01:19:47 INFO [http_cmd]: http status code OK 01:19:47 DEBUG { 01:19:47 DEBUG --- status code --- 01:19:47 DEBUG "status": "success", 01:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:47 DEBUG "data": { 01:19:47 DEBUG --- http response --- 01:19:47 DEBUG "resultType": "vector", 01:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:19:47 DEBUG "result": [ 01:19:47 DEBUG ----- output ----- 01:19:47 DEBUG { 01:19:47 DEBUG { 01:19:47 DEBUG "metric": { 01:19:47 DEBUG "status": "success", 01:19:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:19:47 DEBUG "data": { 01:19:47 DEBUG }, 01:19:47 DEBUG "resultType": "vector", 01:19:47 DEBUG "value": [ 01:19:47 DEBUG "result": [ 01:19:47 DEBUG 01:19:47 DEBUG { 01:19:47 DEBUG 01:19:47 DEBUG "metric": { 01:19:47 DEBUG 01:19:47 DEBUG "pod": "ds-cts-0" 01:19:47 DEBUG [...] 01:19:47 DEBUG }, 01:19:47 DEBUG 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:19:47 DEBUG 01:19:47 DEBUG }, 01:19:47 DEBUG [...] 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 01:19:47 DEBUG 1713485987, 01:19:47 DEBUG 01:19:47 DEBUG "20.076" 01:19:47 DEBUG 01:19:47 DEBUG ] 01:19:47 DEBUG "pod": "ds-cts-2" 01:19:47 DEBUG } 01:19:47 DEBUG }, 01:19:47 DEBUG ] 01:19:47 DEBUG "value": [ 01:19:47 DEBUG } 01:19:47 DEBUG 1713485987, 01:19:47 DEBUG } 01:19:47 DEBUG "0.0003333333333333333" 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG } 01:19:47 INFO [http_cmd]: http status code OK 01:19:47 DEBUG --- status code --- 01:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:47 DEBUG --- http response --- 01:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:47 DEBUG ----- output ----- 01:19:47 DEBUG { 01:19:47 DEBUG "status": "success", 01:19:47 DEBUG "data": { 01:19:47 DEBUG "resultType": "vector", 01:19:47 DEBUG "result": [ 01:19:47 DEBUG { 01:19:47 DEBUG "metric": { 01:19:47 DEBUG "instance": "10.142.0.107:9100" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG [...] 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG "instance": "10.142.0.43:9100" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 1713485987, 01:19:47 DEBUG "13279452.333333332" 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG } 01:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713485987" 01:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713485987" 01:19:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713485987" 01:19:47 INFO [http_cmd]: http status code OK 01:19:47 INFO [http_cmd]: http status code OK 01:19:47 DEBUG --- status code --- 01:19:47 DEBUG --- status code --- 01:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:47 DEBUG --- http response --- 01:19:47 DEBUG --- http response --- 01:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:47 DEBUG ----- output ----- 01:19:47 DEBUG { 01:19:47 DEBUG "status": "success", 01:19:47 DEBUG "data": { 01:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:19:47 DEBUG "resultType": "vector", 01:19:47 DEBUG ----- output ----- 01:19:47 DEBUG "result": [ 01:19:47 DEBUG { 01:19:47 DEBUG { 01:19:47 DEBUG "status": "success", 01:19:47 DEBUG "metric": { 01:19:47 DEBUG "data": { 01:19:47 DEBUG "instance": "10.142.0.107:9100" 01:19:47 DEBUG "resultType": "vector", 01:19:47 DEBUG }, 01:19:47 DEBUG "result": [ 01:19:47 DEBUG "value": [ 01:19:47 DEBUG { 01:19:47 DEBUG 01:19:47 DEBUG "metric": { 01:19:47 DEBUG 01:19:47 DEBUG "pod": "ds-cts-0" 01:19:47 DEBUG 01:19:47 DEBUG }, 01:19:47 DEBUG [...] 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG "instance": "10.142.0.43:9100" 01:19:47 DEBUG [...] 01:19:47 DEBUG }, 01:19:47 DEBUG 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 01:19:47 DEBUG 1713485987, 01:19:47 DEBUG 01:19:47 DEBUG "0.8783333333333326" 01:19:47 DEBUG "pod": "ds-cts-2" 01:19:47 DEBUG ] 01:19:47 DEBUG }, 01:19:47 DEBUG } 01:19:47 DEBUG "value": [ 01:19:47 DEBUG ] 01:19:47 DEBUG 1713485987, 01:19:47 DEBUG } 01:19:47 DEBUG "914.1" 01:19:47 DEBUG } 01:19:47 DEBUG ] 01:19:47 INFO [http_cmd]: http status code OK 01:19:47 DEBUG } 01:19:47 DEBUG --- status code --- 01:19:47 DEBUG ] 01:19:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:19:47 DEBUG } 01:19:47 DEBUG --- http response --- 01:19:47 DEBUG } 01:19:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:19:47 DEBUG ----- output ----- 01:19:47 DEBUG { 01:19:47 DEBUG "status": "success", 01:19:47 DEBUG "data": { 01:19:47 DEBUG "resultType": "vector", 01:19:47 DEBUG "result": [ 01:19:47 DEBUG { 01:19:47 DEBUG "metric": { 01:19:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG [...] 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG 01:19:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:19:47 DEBUG }, 01:19:47 DEBUG "value": [ 01:19:47 DEBUG 1713485987, 01:19:47 DEBUG "116043776" 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG ] 01:19:47 DEBUG } 01:19:47 DEBUG } 01:19:48 INFO 01:19:48 INFO [loop_until]: kubectl --namespace=xlou top pods 01:19:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:19:48 INFO [loop_until]: OK (rc = 0) 01:19:48 DEBUG --- stdout --- 01:19:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 5469m 9449Mi am-6fd554f8f7-hvvkw 4072m 8903Mi am-6fd554f8f7-pmj4b 7074m 9084Mi ds-cts-0 11609m 12650Mi ds-cts-1 10910m 12539Mi ds-cts-2 10835m 12604Mi ds-idrepo-0 2652m 3248Mi ds-idrepo-1 1553m 3167Mi ds-idrepo-2 1701m 3043Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1171Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 1m 64Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 4547m 15Mi 01:19:48 DEBUG --- stderr --- 01:19:48 DEBUG 01:19:48 INFO 01:19:48 INFO [loop_until]: kubectl --namespace=xlou top node 01:19:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:19:48 INFO [loop_until]: OK (rc = 0) 01:19:48 DEBUG --- stdout --- 01:19:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5918Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2103m 7% 6655Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 4021m 13% 11656Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 83m 0% 3237Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 10360m 34% 14915Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5219Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 5219m 17% 12255Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 5382m 18% 3177Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 5228m 17% 11316Mi 10% gke-xlou-cdm-ds-0f601751-j488 1755m 5% 6096Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 10140m 33% 16546Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 2240m 7% 6798Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 11345m 38% 14503Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 3883m 24% 3805Mi 6% 01:19:48 DEBUG --- stderr --- 01:19:48 DEBUG 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713486046" 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713486046" 01:20:46 ERROR [http_cmd]: ERROR 01:20:46 ERROR --- status code --- 01:20:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:20:46 ERROR --- http response --- 01:20:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:20:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713486046" 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486046" 01:20:46 INFO [http_cmd]: http status code OK 01:20:46 DEBUG --- status code --- 01:20:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:46 DEBUG --- http response --- 01:20:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:46 DEBUG ----- output ----- 01:20:46 DEBUG { 01:20:46 DEBUG "status": "success", 01:20:46 DEBUG "data": { 01:20:46 DEBUG "resultType": "vector", 01:20:46 DEBUG "result": [ 01:20:46 DEBUG { 01:20:46 DEBUG "metric": { 01:20:46 DEBUG "instance": "10.142.0.107:9100" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG [...] 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG "instance": "10.142.0.43:9100" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 1713486046, 01:20:46 DEBUG "0.7605333333333372" 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:46 INFO [http_cmd]: http status code OK 01:20:46 DEBUG --- status code --- 01:20:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:46 DEBUG --- http response --- 01:20:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:46 DEBUG ----- output ----- 01:20:46 DEBUG { 01:20:46 DEBUG "status": "success", 01:20:46 DEBUG "data": { 01:20:46 DEBUG "resultType": "vector", 01:20:46 DEBUG "result": [ 01:20:46 DEBUG { 01:20:46 DEBUG "metric": { 01:20:46 DEBUG "instance": "10.142.0.107:9100" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG [...] 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG "instance": "10.142.0.43:9100" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 1713486046, 01:20:46 DEBUG "354601419776" 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:46 INFO [http_cmd]: http status code OK 01:20:46 DEBUG --- status code --- 01:20:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:46 DEBUG --- http response --- 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486046" 01:20:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:46 DEBUG ----- output ----- 01:20:46 DEBUG { 01:20:46 DEBUG "status": "success", 01:20:46 DEBUG "data": { 01:20:46 DEBUG "resultType": "vector", 01:20:46 DEBUG "result": [ 01:20:46 DEBUG { 01:20:46 DEBUG "metric": { 01:20:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG [...] 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG "pod": "ds-cts-0" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 1713486046, 01:20:46 DEBUG "4489435.201485133" 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486046" 01:20:46 INFO [http_cmd]: http status code OK 01:20:46 DEBUG --- status code --- 01:20:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:46 DEBUG --- http response --- 01:20:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:20:46 INFO [http_cmd]: http status code OK 01:20:46 DEBUG --- status code --- 01:20:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:46 DEBUG --- http response --- 01:20:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:46 DEBUG ----- output ----- 01:20:46 DEBUG { 01:20:46 DEBUG "status": "success", 01:20:46 DEBUG "data": { 01:20:46 DEBUG "resultType": "vector", 01:20:46 DEBUG "result": [ 01:20:46 DEBUG { 01:20:46 DEBUG "metric": { 01:20:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG [...] 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 1713486046, 01:20:46 DEBUG "1910.7666666666667" 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713486046" 01:20:46 INFO [http_cmd]: http status code OK 01:20:46 DEBUG --- status code --- 01:20:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:46 DEBUG --- http response --- 01:20:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:20:46 DEBUG ----- output ----- 01:20:46 DEBUG { 01:20:46 DEBUG "status": "success", 01:20:46 DEBUG "data": { 01:20:46 DEBUG "resultType": "vector", 01:20:46 DEBUG "result": [ 01:20:46 DEBUG { 01:20:46 DEBUG "metric": { 01:20:46 DEBUG "pod": "ds-idrepo-0" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG [...] 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG "pod": "ds-cts-2" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 1713486046, 01:20:46 DEBUG "0" 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713486046" 01:20:46 INFO [http_cmd]: http status code OK 01:20:46 DEBUG --- status code --- 01:20:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:46 DEBUG --- http response --- 01:20:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:46 DEBUG ----- output ----- 01:20:46 DEBUG { 01:20:46 DEBUG "status": "success", 01:20:46 DEBUG "data": { 01:20:46 DEBUG "resultType": "vector", 01:20:46 DEBUG "result": [ 01:20:46 DEBUG { 01:20:46 DEBUG "metric": { 01:20:46 DEBUG "instance": "10.142.0.107:9100" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG [...] 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG "instance": "10.142.0.43:9100" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 1713486046, 01:20:46 DEBUG "13002866.2" 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486046" 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486046" 01:20:46 INFO [http_cmd]: http status code OK 01:20:46 DEBUG --- status code --- 01:20:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:46 DEBUG --- http response --- 01:20:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:46 DEBUG ----- output ----- 01:20:46 DEBUG { 01:20:46 DEBUG "status": "success", 01:20:46 DEBUG "data": { 01:20:46 DEBUG "resultType": "vector", 01:20:46 DEBUG "result": [ 01:20:46 DEBUG { 01:20:46 DEBUG "metric": { 01:20:46 DEBUG "pod": "ds-idrepo-0" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG [...] 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG "pod": "ds-idrepo-2" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 1713486046, 01:20:46 DEBUG "0.06666666666666667" 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:46 INFO [http_cmd]: http status code OK 01:20:46 DEBUG --- status code --- 01:20:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:46 DEBUG --- http response --- 01:20:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:20:46 DEBUG ----- output ----- 01:20:46 DEBUG { 01:20:46 DEBUG "status": "success", 01:20:46 DEBUG "data": { 01:20:46 DEBUG "resultType": "vector", 01:20:46 DEBUG "result": [ 01:20:46 DEBUG { 01:20:46 DEBUG "metric": { 01:20:46 DEBUG "pod": "ds-cts-0" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG [...] 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG "pod": "ds-cts-2" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 1713486046, 01:20:46 DEBUG "0" 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486046" 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486046" 01:20:46 INFO [http_cmd]: http status code OK 01:20:46 INFO [http_cmd]: http status code OK 01:20:46 DEBUG --- status code --- 01:20:46 DEBUG --- status code --- 01:20:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:46 DEBUG --- http response --- 01:20:46 DEBUG --- http response --- 01:20:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:46 DEBUG ----- output ----- 01:20:46 DEBUG ----- output ----- 01:20:46 DEBUG { 01:20:46 DEBUG { 01:20:46 DEBUG "status": "success", 01:20:46 DEBUG "status": "success", 01:20:46 DEBUG "data": { 01:20:46 DEBUG "data": { 01:20:46 DEBUG "resultType": "vector", 01:20:46 DEBUG "resultType": "vector", 01:20:46 DEBUG "result": [ 01:20:46 DEBUG "result": [ 01:20:46 DEBUG { 01:20:46 DEBUG { 01:20:46 DEBUG "metric": { 01:20:46 DEBUG "metric": { 01:20:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:20:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:20:46 DEBUG }, 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG [...] 01:20:46 DEBUG [...] 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG "pod": "ds-cts-0" 01:20:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:20:46 DEBUG }, 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 1713486046, 01:20:46 DEBUG 1713486046, 01:20:46 DEBUG "12993718.656979164" 01:20:46 DEBUG "0.06666666666666667" 01:20:46 DEBUG ] 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:46 DEBUG ] 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486046" 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713486046" 01:20:46 INFO [http_cmd]: http status code OK 01:20:46 DEBUG --- status code --- 01:20:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:46 DEBUG --- http response --- 01:20:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:46 DEBUG ----- output ----- 01:20:46 DEBUG { 01:20:46 DEBUG "status": "success", 01:20:46 DEBUG "data": { 01:20:46 DEBUG "resultType": "vector", 01:20:46 DEBUG "result": [ 01:20:46 DEBUG { 01:20:46 DEBUG "metric": { 01:20:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG [...] 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 1713486046, 01:20:46 DEBUG "0" 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:46 INFO [http_cmd]: http status code OK 01:20:46 DEBUG --- status code --- 01:20:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:46 DEBUG --- http response --- 01:20:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:46 DEBUG ----- output ----- 01:20:46 DEBUG { 01:20:46 DEBUG "status": "success", 01:20:46 DEBUG "data": { 01:20:46 DEBUG "resultType": "vector", 01:20:46 DEBUG "result": [ 01:20:46 DEBUG { 01:20:46 DEBUG "metric": { 01:20:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG [...] 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 1713486046, 01:20:46 DEBUG "0.5166151727067788" 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486046" 01:20:46 INFO [http_cmd]: http status code OK 01:20:46 DEBUG --- status code --- 01:20:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:46 DEBUG --- http response --- 01:20:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:46 DEBUG ----- output ----- 01:20:46 DEBUG { 01:20:46 DEBUG "status": "success", 01:20:46 DEBUG "data": { 01:20:46 DEBUG "resultType": "vector", 01:20:46 DEBUG "result": [ 01:20:46 DEBUG { 01:20:46 DEBUG "metric": { 01:20:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG [...] 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG 01:20:46 DEBUG "pod": "ds-cts-0" 01:20:46 DEBUG }, 01:20:46 DEBUG "value": [ 01:20:46 DEBUG 1713486046, 01:20:46 DEBUG "8.020162948500417" 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG ] 01:20:46 DEBUG } 01:20:46 DEBUG } 01:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486047" 01:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486047" 01:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486047" 01:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486047" 01:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486047" 01:20:47 INFO [http_cmd]: http status code OK 01:20:47 DEBUG --- status code --- 01:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:47 DEBUG --- http response --- 01:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 01:20:47 DEBUG ----- output ----- 01:20:47 DEBUG { 01:20:47 DEBUG "status": "success", 01:20:47 DEBUG "data": { 01:20:47 DEBUG "resultType": "vector", 01:20:47 DEBUG "result": [ 01:20:47 DEBUG { 01:20:47 DEBUG "metric": { 01:20:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG [...] 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG "pod": "ds-cts-0" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 1713486047, 01:20:47 DEBUG "28941675.511314537" 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:47 INFO [http_cmd]: http status code OK 01:20:47 INFO [http_cmd]: http status code OK 01:20:47 DEBUG --- status code --- 01:20:47 INFO [http_cmd]: http status code OK 01:20:47 DEBUG --- status code --- 01:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:47 DEBUG --- status code --- 01:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:47 DEBUG --- http response --- 01:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:47 DEBUG --- http response --- 01:20:47 DEBUG --- http response --- 01:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:47 DEBUG ----- output ----- 01:20:47 DEBUG ----- output ----- 01:20:47 DEBUG ----- output ----- 01:20:47 DEBUG { 01:20:47 DEBUG { 01:20:47 DEBUG { 01:20:47 DEBUG "status": "success", 01:20:47 DEBUG "status": "success", 01:20:47 DEBUG "status": "success", 01:20:47 DEBUG "data": { 01:20:47 DEBUG "data": { 01:20:47 DEBUG "data": { 01:20:47 DEBUG "resultType": "vector", 01:20:47 DEBUG "resultType": "vector", 01:20:47 DEBUG "resultType": "vector", 01:20:47 DEBUG "result": [ 01:20:47 DEBUG "result": [ 01:20:47 INFO [http_cmd]: http status code OK 01:20:47 DEBUG "result": [ 01:20:47 DEBUG { 01:20:47 DEBUG { 01:20:47 DEBUG --- status code --- 01:20:47 DEBUG { 01:20:47 DEBUG "metric": { 01:20:47 DEBUG "metric": { 01:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:47 DEBUG "metric": { 01:20:47 DEBUG "pod": "ds-idrepo-0" 01:20:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:20:47 DEBUG --- http response --- 01:20:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:20:47 DEBUG }, 01:20:47 DEBUG }, 01:20:47 DEBUG }, 01:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:47 DEBUG "value": [ 01:20:47 DEBUG "value": [ 01:20:47 DEBUG "value": [ 01:20:47 DEBUG ----- output ----- 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG { 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG "status": "success", 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG "data": { 01:20:47 DEBUG [...] 01:20:47 DEBUG [...] 01:20:47 DEBUG [...] 01:20:47 DEBUG "resultType": "vector", 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG "result": [ 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG { 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG "metric": { 01:20:47 DEBUG "pod": "ds-idrepo-2" 01:20:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:20:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:20:47 DEBUG "pod": "ds-idrepo-0" 01:20:47 DEBUG }, 01:20:47 DEBUG }, 01:20:47 DEBUG }, 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG "value": [ 01:20:47 DEBUG "value": [ 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 1713486047, 01:20:47 DEBUG 1713486047, 01:20:47 DEBUG 1713486047, 01:20:47 DEBUG 01:20:47 DEBUG "0.06666666666666667" 01:20:47 DEBUG "0.10516222222222221" 01:20:47 DEBUG "0.06666666666666667" 01:20:47 DEBUG 01:20:47 DEBUG ] 01:20:47 DEBUG ] 01:20:47 DEBUG ] 01:20:47 DEBUG 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:47 DEBUG [...] 01:20:47 DEBUG ] 01:20:47 DEBUG ] 01:20:47 DEBUG ] 01:20:47 DEBUG 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:47 DEBUG 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:47 DEBUG 01:20:47 DEBUG "pod": "ds-idrepo-2" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 1713486047, 01:20:47 DEBUG "0.06666666666666667" 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486047" 01:20:47 INFO [http_cmd]: http status code OK 01:20:47 DEBUG --- status code --- 01:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:47 DEBUG --- http response --- 01:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:47 DEBUG ----- output ----- 01:20:47 DEBUG { 01:20:47 DEBUG "status": "success", 01:20:47 DEBUG "data": { 01:20:47 DEBUG "resultType": "vector", 01:20:47 DEBUG "result": [ 01:20:47 DEBUG { 01:20:47 DEBUG "metric": { 01:20:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG [...] 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 1713486047, 01:20:47 DEBUG "2997920" 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486047" 01:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713486047" 01:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713486047" 01:20:47 INFO [http_cmd]: http status code OK 01:20:47 DEBUG --- status code --- 01:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:47 DEBUG --- http response --- 01:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:20:47 DEBUG ----- output ----- 01:20:47 DEBUG { 01:20:47 DEBUG "status": "success", 01:20:47 DEBUG "data": { 01:20:47 DEBUG "resultType": "vector", 01:20:47 DEBUG "result": [ 01:20:47 DEBUG { 01:20:47 DEBUG "metric": { 01:20:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG [...] 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 1713486047, 01:20:47 DEBUG "20.076" 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:47 INFO [http_cmd]: http status code OK 01:20:47 DEBUG --- status code --- 01:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:47 DEBUG --- http response --- 01:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:20:47 DEBUG ----- output ----- 01:20:47 DEBUG { 01:20:47 DEBUG "status": "success", 01:20:47 DEBUG "data": { 01:20:47 DEBUG "resultType": "vector", 01:20:47 DEBUG "result": [ 01:20:47 DEBUG { 01:20:47 DEBUG "metric": { 01:20:47 DEBUG "pod": "ds-cts-0" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG [...] 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG "pod": "ds-cts-2" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 1713486047, 01:20:47 DEBUG "0.00012643111111111113" 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:47 INFO [http_cmd]: http status code OK 01:20:47 DEBUG --- status code --- 01:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:47 DEBUG --- http response --- 01:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:47 DEBUG ----- output ----- 01:20:47 DEBUG { 01:20:47 DEBUG "status": "success", 01:20:47 DEBUG "data": { 01:20:47 DEBUG "resultType": "vector", 01:20:47 DEBUG "result": [ 01:20:47 DEBUG { 01:20:47 DEBUG "metric": { 01:20:47 DEBUG "instance": "10.142.0.107:9100" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG [...] 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG "instance": "10.142.0.43:9100" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 1713486047, 01:20:47 DEBUG "13012108.633333333" 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713486047" 01:20:47 INFO [http_cmd]: http status code OK 01:20:47 DEBUG --- status code --- 01:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:47 DEBUG --- http response --- 01:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:47 DEBUG ----- output ----- 01:20:47 DEBUG { 01:20:47 DEBUG "status": "success", 01:20:47 DEBUG "data": { 01:20:47 DEBUG "resultType": "vector", 01:20:47 DEBUG "result": [ 01:20:47 DEBUG { 01:20:47 DEBUG "metric": { 01:20:47 DEBUG "instance": "10.142.0.107:9100" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG [...] 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG "instance": "10.142.0.43:9100" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 1713486047, 01:20:47 DEBUG "0.7823333333333334" 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486047" 01:20:47 INFO [http_cmd]: http status code OK 01:20:47 DEBUG --- status code --- 01:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:47 DEBUG --- http response --- 01:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:20:47 DEBUG ----- output ----- 01:20:47 DEBUG { 01:20:47 DEBUG "status": "success", 01:20:47 DEBUG "data": { 01:20:47 DEBUG "resultType": "vector", 01:20:47 DEBUG "result": [ 01:20:47 DEBUG { 01:20:47 DEBUG "metric": { 01:20:47 DEBUG "pod": "ds-cts-0" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG [...] 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG "pod": "ds-cts-2" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 1713486047, 01:20:47 DEBUG "542.9" 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713486047" 01:20:47 INFO [http_cmd]: http status code OK 01:20:47 DEBUG --- status code --- 01:20:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:20:47 DEBUG --- http response --- 01:20:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:20:47 DEBUG ----- output ----- 01:20:47 DEBUG { 01:20:47 DEBUG "status": "success", 01:20:47 DEBUG "data": { 01:20:47 DEBUG "resultType": "vector", 01:20:47 DEBUG "result": [ 01:20:47 DEBUG { 01:20:47 DEBUG "metric": { 01:20:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG [...] 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG 01:20:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:20:47 DEBUG }, 01:20:47 DEBUG "value": [ 01:20:47 DEBUG 1713486047, 01:20:47 DEBUG "385748992" 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG ] 01:20:47 DEBUG } 01:20:47 DEBUG } 01:20:48 INFO 01:20:48 INFO [loop_until]: kubectl --namespace=xlou top pods 01:20:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:20:48 INFO [loop_until]: OK (rc = 0) 01:20:48 DEBUG --- stdout --- 01:20:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2231m 9366Mi am-6fd554f8f7-hvvkw 1848m 8832Mi am-6fd554f8f7-pmj4b 1990m 9026Mi ds-cts-0 7439m 12781Mi ds-cts-1 8235m 12514Mi ds-cts-2 7864m 12844Mi ds-idrepo-0 861m 3270Mi ds-idrepo-1 996m 3173Mi ds-idrepo-2 568m 3047Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 6m 1173Mi idm-b85f9d978-wnhxb 7m 1215Mi lodemon-5d46798758-5h2st 8m 64Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 491m 274Mi 01:20:48 DEBUG --- stderr --- 01:20:48 DEBUG 01:20:48 INFO 01:20:48 INFO [loop_until]: kubectl --namespace=xlou top node 01:20:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:20:48 INFO [loop_until]: OK (rc = 0) 01:20:48 DEBUG --- stdout --- 01:20:48 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 75m 0% 5933Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 968m 3% 6627Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1845m 6% 11611Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 118m 0% 3279Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8059m 26% 15524Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 168m 0% 5205Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2352m 7% 12190Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 95m 0% 3125Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2049m 6% 11279Mi 10% gke-xlou-cdm-ds-0f601751-j488 946m 3% 6059Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8480m 28% 16593Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 919m 3% 6813Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8000m 26% 14733Mi 13% gke-xlou-cdm-frontend-a8771548-i82n 747m 4% 4059Mi 6% 01:20:48 DEBUG --- stderr --- 01:20:48 DEBUG 01:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713486106" 01:21:46 ERROR [http_cmd]: ERROR 01:21:46 ERROR --- status code --- 01:21:46 ERROR http status code is 400 (expected [200, 401, 403, 404]) 01:21:46 ERROR --- http response --- 01:21:46 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 01:21:46 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 01:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713486106" 01:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713486106" 01:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486106" 01:21:46 INFO [http_cmd]: http status code OK 01:21:46 INFO [http_cmd]: http status code OK 01:21:46 INFO [http_cmd]: http status code OK 01:21:46 DEBUG --- status code --- 01:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486106" 01:21:46 DEBUG --- status code --- 01:21:46 DEBUG --- status code --- 01:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:46 DEBUG --- http response --- 01:21:46 DEBUG --- http response --- 01:21:46 DEBUG --- http response --- 01:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:46 DEBUG ----- output ----- 01:21:46 DEBUG ----- output ----- 01:21:46 DEBUG ----- output ----- 01:21:46 DEBUG { 01:21:46 DEBUG { 01:21:46 DEBUG { 01:21:46 DEBUG "status": "success", 01:21:46 DEBUG "status": "success", 01:21:46 DEBUG "status": "success", 01:21:46 DEBUG "data": { 01:21:46 DEBUG "data": { 01:21:46 DEBUG "data": { 01:21:46 DEBUG "resultType": "vector", 01:21:46 DEBUG "resultType": "vector", 01:21:46 DEBUG "resultType": "vector", 01:21:46 DEBUG "result": [ 01:21:46 DEBUG "result": [ 01:21:46 DEBUG "result": [ 01:21:46 DEBUG { 01:21:46 DEBUG { 01:21:46 DEBUG { 01:21:46 DEBUG "metric": { 01:21:46 DEBUG "metric": { 01:21:46 DEBUG "metric": { 01:21:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:21:46 DEBUG "instance": "10.142.0.107:9100" 01:21:46 DEBUG "instance": "10.142.0.107:9100" 01:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486106" 01:21:46 DEBUG }, 01:21:46 DEBUG }, 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG "value": [ 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG [...] 01:21:46 DEBUG [...] 01:21:46 DEBUG [...] 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG "pod": "ds-cts-2" 01:21:46 DEBUG "instance": "10.142.0.43:9100" 01:21:46 DEBUG "instance": "10.142.0.43:9100" 01:21:46 DEBUG }, 01:21:46 DEBUG }, 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG "value": [ 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 1713486106, 01:21:46 DEBUG 1713486106, 01:21:46 DEBUG 1713486106, 01:21:46 DEBUG "9720897.803876938" 01:21:46 DEBUG "355199673344" 01:21:46 DEBUG "0.6767333333333236" 01:21:46 DEBUG ] 01:21:46 DEBUG ] 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:46 DEBUG ] 01:21:46 DEBUG ] 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:46 INFO [http_cmd]: http status code OK 01:21:46 DEBUG --- status code --- 01:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:46 DEBUG --- http response --- 01:21:46 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 01:21:46 INFO [http_cmd]: http status code OK 01:21:46 DEBUG --- status code --- 01:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:46 DEBUG --- http response --- 01:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:46 DEBUG ----- output ----- 01:21:46 DEBUG { 01:21:46 DEBUG "status": "success", 01:21:46 DEBUG "data": { 01:21:46 DEBUG "resultType": "vector", 01:21:46 DEBUG "result": [ 01:21:46 DEBUG { 01:21:46 DEBUG "metric": { 01:21:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG [...] 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 1713486106, 01:21:46 DEBUG "1916.5666666666666" 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713486106" 01:21:46 INFO [http_cmd]: http status code OK 01:21:46 DEBUG --- status code --- 01:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:46 DEBUG --- http response --- 01:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:21:46 DEBUG ----- output ----- 01:21:46 DEBUG { 01:21:46 DEBUG "status": "success", 01:21:46 DEBUG "data": { 01:21:46 DEBUG "resultType": "vector", 01:21:46 DEBUG "result": [ 01:21:46 DEBUG { 01:21:46 DEBUG "metric": { 01:21:46 DEBUG "pod": "ds-idrepo-0" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG [...] 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG "pod": "ds-cts-2" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 1713486106, 01:21:46 DEBUG "0" 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713486106" 01:21:46 INFO [http_cmd]: http status code OK 01:21:46 DEBUG --- status code --- 01:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:46 DEBUG --- http response --- 01:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:46 DEBUG ----- output ----- 01:21:46 DEBUG { 01:21:46 DEBUG "status": "success", 01:21:46 DEBUG "data": { 01:21:46 DEBUG "resultType": "vector", 01:21:46 DEBUG "result": [ 01:21:46 DEBUG { 01:21:46 DEBUG "metric": { 01:21:46 DEBUG "instance": "10.142.0.107:9100" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG [...] 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG "instance": "10.142.0.43:9100" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 1713486106, 01:21:46 DEBUG "12697782.4" 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486106" 01:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486106" 01:21:46 INFO [http_cmd]: http status code OK 01:21:46 DEBUG --- status code --- 01:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:46 DEBUG --- http response --- 01:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:21:46 DEBUG ----- output ----- 01:21:46 DEBUG { 01:21:46 DEBUG "status": "success", 01:21:46 DEBUG "data": { 01:21:46 DEBUG "resultType": "vector", 01:21:46 DEBUG "result": [ 01:21:46 DEBUG { 01:21:46 DEBUG "metric": { 01:21:46 DEBUG "pod": "ds-cts-0" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG [...] 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG "pod": "ds-cts-2" 01:21:46 DEBUG }, 01:21:46 INFO [http_cmd]: http status code OK 01:21:46 DEBUG "value": [ 01:21:46 DEBUG --- status code --- 01:21:46 DEBUG 1713486106, 01:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:46 DEBUG "0" 01:21:46 DEBUG --- http response --- 01:21:46 DEBUG ] 01:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:46 DEBUG } 01:21:46 DEBUG ----- output ----- 01:21:46 DEBUG ] 01:21:46 DEBUG { 01:21:46 DEBUG } 01:21:46 DEBUG "status": "success", 01:21:46 DEBUG } 01:21:46 DEBUG "data": { 01:21:46 DEBUG "resultType": "vector", 01:21:46 DEBUG "result": [ 01:21:46 DEBUG { 01:21:46 DEBUG "metric": { 01:21:46 DEBUG "pod": "ds-idrepo-0" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG [...] 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG "pod": "ds-idrepo-2" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 1713486106, 01:21:46 DEBUG "0" 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486106" 01:21:46 INFO [http_cmd]: http status code OK 01:21:46 DEBUG --- status code --- 01:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:46 DEBUG --- http response --- 01:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:46 DEBUG ----- output ----- 01:21:46 DEBUG { 01:21:46 DEBUG "status": "success", 01:21:46 DEBUG "data": { 01:21:46 DEBUG "resultType": "vector", 01:21:46 DEBUG "result": [ 01:21:46 DEBUG { 01:21:46 DEBUG "metric": { 01:21:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG [...] 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 1713486106, 01:21:46 DEBUG "0" 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486106" 01:21:46 INFO [http_cmd]: http status code OK 01:21:46 DEBUG --- status code --- 01:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:46 DEBUG --- http response --- 01:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486106" 01:21:46 DEBUG ----- output ----- 01:21:46 DEBUG { 01:21:46 DEBUG "status": "success", 01:21:46 DEBUG "data": { 01:21:46 DEBUG "resultType": "vector", 01:21:46 DEBUG "result": [ 01:21:46 DEBUG { 01:21:46 DEBUG "metric": { 01:21:46 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:21:46 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713486106" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG [...] 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG "pod": "ds-cts-2" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 1713486106, 01:21:46 DEBUG "360210210.0964303" 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:46 INFO [http_cmd]: http status code OK 01:21:46 DEBUG --- status code --- 01:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:46 DEBUG --- http response --- 01:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:46 DEBUG ----- output ----- 01:21:46 DEBUG { 01:21:46 DEBUG "status": "success", 01:21:46 DEBUG "data": { 01:21:46 DEBUG "resultType": "vector", 01:21:46 DEBUG "result": [ 01:21:46 DEBUG { 01:21:46 DEBUG "metric": { 01:21:46 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG [...] 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 1713486106, 01:21:46 DEBUG "0" 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:46 INFO [http_cmd]: http status code OK 01:21:46 DEBUG --- status code --- 01:21:46 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:46 DEBUG --- http response --- 01:21:46 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:46 DEBUG ----- output ----- 01:21:46 DEBUG { 01:21:46 DEBUG "status": "success", 01:21:46 DEBUG "data": { 01:21:46 DEBUG "resultType": "vector", 01:21:46 DEBUG "result": [ 01:21:46 DEBUG { 01:21:46 DEBUG "metric": { 01:21:46 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG [...] 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG 01:21:46 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:21:46 DEBUG }, 01:21:46 DEBUG "value": [ 01:21:46 DEBUG 1713486106, 01:21:46 DEBUG "1.0371499653378489" 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG ] 01:21:46 DEBUG } 01:21:46 DEBUG } 01:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486107" 01:21:47 INFO [http_cmd]: http status code OK 01:21:47 DEBUG --- status code --- 01:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:47 DEBUG --- http response --- 01:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 01:21:47 DEBUG ----- output ----- 01:21:47 DEBUG { 01:21:47 DEBUG "status": "success", 01:21:47 DEBUG "data": { 01:21:47 DEBUG "resultType": "vector", 01:21:47 DEBUG "result": [ 01:21:47 DEBUG { 01:21:47 DEBUG "metric": { 01:21:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG [...] 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG "pod": "ds-cts-2" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 1713486107, 01:21:47 DEBUG "3.435617304284517" 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG } 01:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486107" 01:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486107" 01:21:47 INFO [http_cmd]: http status code OK 01:21:47 DEBUG --- status code --- 01:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:47 DEBUG --- http response --- 01:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:47 DEBUG ----- output ----- 01:21:47 DEBUG { 01:21:47 DEBUG "status": "success", 01:21:47 DEBUG "data": { 01:21:47 DEBUG "resultType": "vector", 01:21:47 DEBUG "result": [ 01:21:47 DEBUG { 01:21:47 DEBUG "metric": { 01:21:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG [...] 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 1713486107, 01:21:47 DEBUG "0" 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG } 01:21:47 INFO [http_cmd]: http status code OK 01:21:47 DEBUG --- status code --- 01:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:47 DEBUG --- http response --- 01:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:47 DEBUG ----- output ----- 01:21:47 DEBUG { 01:21:47 DEBUG "status": "success", 01:21:47 DEBUG "data": { 01:21:47 DEBUG "resultType": "vector", 01:21:47 DEBUG "result": [ 01:21:47 DEBUG { 01:21:47 DEBUG "metric": { 01:21:47 DEBUG "pod": "ds-idrepo-0" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG [...] 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG "pod": "ds-idrepo-2" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 1713486107, 01:21:47 DEBUG "0" 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG } 01:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486107" 01:21:47 INFO [http_cmd]: http status code OK 01:21:47 DEBUG --- status code --- 01:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486107" 01:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486107" 01:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:47 DEBUG --- http response --- 01:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:47 DEBUG ----- output ----- 01:21:47 DEBUG { 01:21:47 DEBUG "status": "success", 01:21:47 DEBUG "data": { 01:21:47 DEBUG "resultType": "vector", 01:21:47 DEBUG "result": [ 01:21:47 DEBUG { 01:21:47 DEBUG "metric": { 01:21:47 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG [...] 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG "pod": "ds-cts-2" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 1713486107, 01:21:47 DEBUG "15149647.419141911" 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG } 01:21:47 INFO [http_cmd]: http status code OK 01:21:47 DEBUG --- status code --- 01:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:47 DEBUG --- http response --- 01:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:47 DEBUG ----- output ----- 01:21:47 DEBUG { 01:21:47 DEBUG "status": "success", 01:21:47 DEBUG "data": { 01:21:47 DEBUG "resultType": "vector", 01:21:47 DEBUG "result": [ 01:21:47 DEBUG { 01:21:47 DEBUG "metric": { 01:21:47 DEBUG "pod": "ds-idrepo-0" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG [...] 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG "pod": "ds-idrepo-2" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 1713486107, 01:21:47 DEBUG "0" 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG } 01:21:47 INFO [http_cmd]: http status code OK 01:21:47 DEBUG --- status code --- 01:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:47 DEBUG --- http response --- 01:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:47 DEBUG ----- output ----- 01:21:47 DEBUG { 01:21:47 DEBUG "status": "success", 01:21:47 DEBUG "data": { 01:21:47 DEBUG "resultType": "vector", 01:21:47 DEBUG "result": [ 01:21:47 DEBUG { 01:21:47 DEBUG "metric": { 01:21:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG [...] 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 1713486107, 01:21:47 DEBUG "0" 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG } 01:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486107" 01:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486107" 01:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713486107" 01:21:47 INFO [http_cmd]: http status code OK 01:21:47 INFO [http_cmd]: http status code OK 01:21:47 DEBUG --- status code --- 01:21:47 DEBUG --- status code --- 01:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:47 DEBUG --- http response --- 01:21:47 DEBUG --- http response --- 01:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:21:47 DEBUG ----- output ----- 01:21:47 DEBUG ----- output ----- 01:21:47 DEBUG { 01:21:47 DEBUG { 01:21:47 DEBUG "status": "success", 01:21:47 DEBUG "status": "success", 01:21:47 DEBUG "data": { 01:21:47 DEBUG "data": { 01:21:47 DEBUG "resultType": "vector", 01:21:47 DEBUG "resultType": "vector", 01:21:47 DEBUG "result": [ 01:21:47 DEBUG "result": [ 01:21:47 DEBUG { 01:21:47 DEBUG { 01:21:47 DEBUG "metric": { 01:21:47 DEBUG "metric": { 01:21:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:21:47 DEBUG "pod": "ds-cts-0" 01:21:47 DEBUG }, 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG [...] 01:21:47 DEBUG [...] 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:21:47 DEBUG "pod": "ds-cts-2" 01:21:47 DEBUG }, 01:21:47 DEBUG }, 01:21:47 INFO [http_cmd]: http status code OK 01:21:47 DEBUG "value": [ 01:21:47 DEBUG "value": [ 01:21:47 DEBUG --- status code --- 01:21:47 DEBUG 1713486107, 01:21:47 DEBUG 1713486107, 01:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:47 DEBUG "2883004" 01:21:47 DEBUG "0.00013333333333333334" 01:21:47 DEBUG --- http response --- 01:21:47 DEBUG ] 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:47 DEBUG } 01:21:47 DEBUG ] 01:21:47 DEBUG ----- output ----- 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG { 01:21:47 DEBUG } 01:21:47 DEBUG } 01:21:47 DEBUG "status": "success", 01:21:47 DEBUG } 01:21:47 DEBUG "data": { 01:21:47 DEBUG "resultType": "vector", 01:21:47 DEBUG "result": [ 01:21:47 DEBUG { 01:21:47 DEBUG "metric": { 01:21:47 DEBUG "instance": "10.142.0.107:9100" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG [...] 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG "instance": "10.142.0.43:9100" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 1713486107, 01:21:47 DEBUG "12706702.166666666" 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG } 01:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713486107" 01:21:47 INFO [http_cmd]: http status code OK 01:21:47 DEBUG --- status code --- 01:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:47 DEBUG --- http response --- 01:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 01:21:47 DEBUG ----- output ----- 01:21:47 DEBUG { 01:21:47 DEBUG "status": "success", 01:21:47 DEBUG "data": { 01:21:47 DEBUG "resultType": "vector", 01:21:47 DEBUG "result": [ 01:21:47 DEBUG { 01:21:47 DEBUG "metric": { 01:21:47 DEBUG "pod": "am-6fd554f8f7-hvvkw" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG [...] 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG "pod": "am-6fd554f8f7-pmj4b" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 1713486107, 01:21:47 DEBUG "20.076" 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG } 01:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713486107" 01:21:47 INFO [http_cmd]: http status code OK 01:21:47 DEBUG --- status code --- 01:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:47 DEBUG --- http response --- 01:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:47 DEBUG ----- output ----- 01:21:47 DEBUG { 01:21:47 DEBUG "status": "success", 01:21:47 DEBUG "data": { 01:21:47 DEBUG "resultType": "vector", 01:21:47 DEBUG "result": [ 01:21:47 DEBUG { 01:21:47 DEBUG "metric": { 01:21:47 DEBUG "instance": "10.142.0.107:9100" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG [...] 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG "instance": "10.142.0.43:9100" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 1713486107, 01:21:47 DEBUG "0.6543333333333323" 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG } 01:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713486107" 01:21:47 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713486107" 01:21:47 INFO [http_cmd]: http status code OK 01:21:47 DEBUG --- status code --- 01:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:47 DEBUG --- http response --- 01:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 01:21:47 DEBUG ----- output ----- 01:21:47 DEBUG { 01:21:47 DEBUG "status": "success", 01:21:47 DEBUG "data": { 01:21:47 DEBUG "resultType": "vector", 01:21:47 DEBUG "result": [ 01:21:47 DEBUG { 01:21:47 DEBUG "metric": { 01:21:47 DEBUG "pod": "ds-cts-0" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG [...] 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG "pod": "ds-cts-2" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 1713486107, 01:21:47 DEBUG "437.9" 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG } 01:21:47 INFO [http_cmd]: http status code OK 01:21:47 DEBUG --- status code --- 01:21:47 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 01:21:47 DEBUG --- http response --- 01:21:47 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 01:21:47 DEBUG ----- output ----- 01:21:47 DEBUG { 01:21:47 DEBUG "status": "success", 01:21:47 DEBUG "data": { 01:21:47 DEBUG "resultType": "vector", 01:21:47 DEBUG "result": [ 01:21:47 DEBUG { 01:21:47 DEBUG "metric": { 01:21:47 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG [...] 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG 01:21:47 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 01:21:47 DEBUG }, 01:21:47 DEBUG "value": [ 01:21:47 DEBUG 1713486107, 01:21:47 DEBUG "1386270720" 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG ] 01:21:47 DEBUG } 01:21:47 DEBUG } 01:21:48 INFO 01:21:48 INFO [loop_until]: kubectl --namespace=xlou top pods 01:21:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:21:48 INFO [loop_until]: OK (rc = 0) 01:21:48 DEBUG --- stdout --- 01:21:48 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6fd554f8f7-6rhpf 2220m 9364Mi am-6fd554f8f7-hvvkw 1801m 8824Mi am-6fd554f8f7-pmj4b 1925m 9020Mi ds-cts-0 8336m 12640Mi ds-cts-1 8966m 12817Mi ds-cts-2 8174m 12807Mi ds-idrepo-0 773m 3276Mi ds-idrepo-1 583m 3182Mi ds-idrepo-2 647m 3060Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1172Mi idm-b85f9d978-wnhxb 3m 1215Mi lodemon-5d46798758-5h2st 5m 64Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 1022m 1227Mi 01:21:48 DEBUG --- stderr --- 01:21:48 DEBUG 01:21:49 INFO 01:21:49 INFO [loop_until]: kubectl --namespace=xlou top node 01:21:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 01:21:49 INFO [loop_until]: OK (rc = 0) 01:21:49 DEBUG --- stdout --- 01:21:49 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5922Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 793m 2% 6665Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1802m 6% 11594Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 80m 0% 3269Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 8356m 27% 14779Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 5200Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2256m 7% 12174Mi 10% gke-xlou-cdm-ds-0f601751-cf7b 67m 0% 3100Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2021m 6% 11255Mi 10% gke-xlou-cdm-ds-0f601751-j488 605m 2% 6121Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8808m 29% 16788Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 745m 2% 6827Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8074m 27% 14257Mi 12% gke-xlou-cdm-frontend-a8771548-i82n 1078m 6% 5114Mi 8% 01:21:49 DEBUG --- stderr --- 01:21:49 DEBUG 01:22:39 INFO Finished: True 01:22:39 INFO Waiting for threads to register finish flag 01:22:49 INFO Done. Have a nice day! :) 127.0.0.1 - - [19/Apr/2024 01:22:49] "GET /monitoring/stop HTTP/1.1" 200 - 01:22:52 WARNING File /tmp/lodemon_data-20240419_004139/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6481', ...) 127.0.0.1 - - [19/Apr/2024 01:22:57] "GET /monitoring/process HTTP/1.1" 200 -