==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-69554696-dct8l Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-1b0769c8-7vmj/10.142.0.111 Start Time: Fri, 19 Apr 2024 15:07:00 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=69554696 skaffold.dev/run-id=1a6b38d0-c6b8-4feb-b0ad-be18126cd04c Annotations: Status: Running IP: 10.106.42.186 IPs: IP: 10.106.42.186 Controlled By: ReplicaSet/lodemon-69554696 Containers: lodemon: Container ID: containerd://80d4022bd87d7cb15c0046bbb665d59bb3312d2bf0a57555ca7745614f615698 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:03f76e6326d713f4d73dc474e0e25fbbbaecc7c1d8c0590bfd94d68ac0d98c97 Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Fri, 19 Apr 2024 15:07:01 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-58mwd (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-58mwd: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 14m default-scheduler Successfully assigned xlou/lodemon-69554696-dct8l to gke-xlou-cdm-default-pool-1b0769c8-7vmj Normal Pulling 14m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 14m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 220.415612ms (220.421362ms including waiting) Normal Created 14m kubelet Created container lodemon Normal Started 14m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.42.186:8080 Press CTRL+C to quit 16:08:08 INFO 16:08:08 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:08 INFO [loop_until]: OK (rc = 0) 16:08:08 DEBUG --- stdout --- 16:08:08 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:08 DEBUG --- stderr --- 16:08:08 DEBUG 16:08:08 INFO 16:08:08 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:08 INFO [loop_until]: OK (rc = 0) 16:08:08 DEBUG --- stdout --- 16:08:08 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:08 DEBUG --- stderr --- 16:08:08 DEBUG 16:08:08 INFO 16:08:08 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:08 INFO [loop_until]: OK (rc = 0) 16:08:08 DEBUG --- stdout --- 16:08:08 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:08 DEBUG --- stderr --- 16:08:08 DEBUG 16:08:08 INFO 16:08:08 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:08 INFO [loop_until]: OK (rc = 0) 16:08:08 DEBUG --- stdout --- 16:08:08 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:08 DEBUG --- stderr --- 16:08:08 DEBUG 16:08:08 INFO 16:08:08 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:08 INFO [loop_until]: OK (rc = 0) 16:08:08 DEBUG --- stdout --- 16:08:08 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:08 DEBUG --- stderr --- 16:08:08 DEBUG 16:08:08 INFO 16:08:08 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:08 INFO [loop_until]: OK (rc = 0) 16:08:08 DEBUG --- stdout --- 16:08:08 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:08 DEBUG --- stderr --- 16:08:08 DEBUG 16:08:08 INFO 16:08:08 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:08 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:09 INFO [loop_until]: OK (rc = 0) 16:08:09 DEBUG --- stdout --- 16:08:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:09 DEBUG --- stderr --- 16:08:09 DEBUG 16:08:09 INFO 16:08:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:09 INFO [loop_until]: OK (rc = 0) 16:08:09 DEBUG --- stdout --- 16:08:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:09 DEBUG --- stderr --- 16:08:09 DEBUG 16:08:09 INFO 16:08:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:09 INFO [loop_until]: OK (rc = 0) 16:08:09 DEBUG --- stdout --- 16:08:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:09 DEBUG --- stderr --- 16:08:09 DEBUG 16:08:09 INFO 16:08:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:09 INFO [loop_until]: OK (rc = 0) 16:08:09 DEBUG --- stdout --- 16:08:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:09 DEBUG --- stderr --- 16:08:09 DEBUG 16:08:09 INFO 16:08:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:09 INFO [loop_until]: OK (rc = 0) 16:08:09 DEBUG --- stdout --- 16:08:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:09 DEBUG --- stderr --- 16:08:09 DEBUG 16:08:09 INFO 16:08:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:09 INFO [loop_until]: OK (rc = 0) 16:08:09 DEBUG --- stdout --- 16:08:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:09 DEBUG --- stderr --- 16:08:09 DEBUG 16:08:09 INFO 16:08:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:09 INFO [loop_until]: OK (rc = 0) 16:08:09 DEBUG --- stdout --- 16:08:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:09 DEBUG --- stderr --- 16:08:09 DEBUG 16:08:09 INFO 16:08:09 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:09 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:09 INFO [loop_until]: OK (rc = 0) 16:08:09 DEBUG --- stdout --- 16:08:09 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:09 DEBUG --- stderr --- 16:08:09 DEBUG 16:08:10 INFO 16:08:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:10 INFO [loop_until]: OK (rc = 0) 16:08:10 DEBUG --- stdout --- 16:08:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:10 DEBUG --- stderr --- 16:08:10 DEBUG 16:08:10 INFO 16:08:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:10 INFO [loop_until]: OK (rc = 0) 16:08:10 DEBUG --- stdout --- 16:08:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:10 DEBUG --- stderr --- 16:08:10 DEBUG 16:08:10 INFO 16:08:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:10 INFO [loop_until]: OK (rc = 0) 16:08:10 DEBUG --- stdout --- 16:08:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:10 DEBUG --- stderr --- 16:08:10 DEBUG 16:08:10 INFO 16:08:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:10 INFO [loop_until]: OK (rc = 0) 16:08:10 DEBUG --- stdout --- 16:08:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:10 DEBUG --- stderr --- 16:08:10 DEBUG 16:08:10 INFO 16:08:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:10 INFO [loop_until]: OK (rc = 0) 16:08:10 DEBUG --- stdout --- 16:08:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:10 DEBUG --- stderr --- 16:08:10 DEBUG 16:08:10 INFO 16:08:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:10 INFO [loop_until]: OK (rc = 0) 16:08:10 DEBUG --- stdout --- 16:08:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:10 DEBUG --- stderr --- 16:08:10 DEBUG 16:08:10 INFO 16:08:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:10 INFO [loop_until]: OK (rc = 0) 16:08:10 DEBUG --- stdout --- 16:08:10 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:10 DEBUG --- stderr --- 16:08:10 DEBUG 16:08:10 INFO 16:08:10 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:10 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:11 INFO [loop_until]: OK (rc = 0) 16:08:11 DEBUG --- stdout --- 16:08:11 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:11 DEBUG --- stderr --- 16:08:11 DEBUG 16:08:11 INFO 16:08:11 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:11 INFO [loop_until]: OK (rc = 0) 16:08:11 DEBUG --- stdout --- 16:08:11 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:11 DEBUG --- stderr --- 16:08:11 DEBUG 16:08:11 INFO 16:08:11 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:11 INFO [loop_until]: OK (rc = 0) 16:08:11 DEBUG --- stdout --- 16:08:11 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:11 DEBUG --- stderr --- 16:08:11 DEBUG 16:08:11 INFO 16:08:11 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:11 INFO [loop_until]: OK (rc = 0) 16:08:11 DEBUG --- stdout --- 16:08:11 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:11 DEBUG --- stderr --- 16:08:11 DEBUG 16:08:11 INFO 16:08:11 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:11 INFO [loop_until]: OK (rc = 0) 16:08:11 DEBUG --- stdout --- 16:08:11 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:11 DEBUG --- stderr --- 16:08:11 DEBUG 16:08:11 INFO 16:08:11 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 16:08:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:11 INFO [loop_until]: OK (rc = 0) 16:08:11 DEBUG --- stdout --- 16:08:11 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 16:08:11 DEBUG --- stderr --- 16:08:11 DEBUG 16:08:11 INFO Initializing monitoring instance threads 16:08:11 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 16:08:11 INFO Starting instance threads 16:08:11 INFO 16:08:11 INFO Thread started 16:08:11 INFO [loop_until]: kubectl --namespace=xlou top node 16:08:11 INFO 16:08:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:11 INFO Thread started 16:08:11 INFO [loop_until]: kubectl --namespace=xlou top pods 16:08:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG --- status code --- 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG --- status code --- 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 INFO Thread started 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG --- status code --- 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG --- http response --- 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 INFO Thread started 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG --- http response --- 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG --- http response --- 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 DEBUG --- http response --- 16:08:11 INFO Thread started 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:08:11 INFO Thread started 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG { 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG --- status code --- 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 DEBUG { 16:08:11 DEBUG ----- output ----- 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 DEBUG --- http response --- 16:08:11 INFO Thread started 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG { 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:08:11 DEBUG { 16:08:11 DEBUG { 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG { 16:08:11 INFO [loop_until]: OK (rc = 0) 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:08:11 INFO [loop_until]: OK (rc = 0) 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG "status": "success", 16:08:11 INFO Thread started 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:08:11 DEBUG { 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:08:11 DEBUG { 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:08:11 DEBUG "data": { 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG "data": { 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG --- stdout --- 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG --- stdout --- 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG "data": { 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 DEBUG --- status code --- 16:08:11 INFO Thread started 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG { 16:08:11 DEBUG "data": { 16:08:11 DEBUG "data": { 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:08:11 DEBUG "data": { 16:08:11 DEBUG { 16:08:11 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 6216Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 67m 0% 6903Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 5260Mi 4% gke-xlou-cdm-default-pool-1b0769c8-qkns 93m 0% 6084Mi 5% gke-xlou-cdm-default-pool-1b0769c8-w15q 62m 0% 15013Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 148m 0% 5201Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 52m 0% 2769Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 83m 0% 4252Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 50m 0% 2177Mi 1% gke-xlou-cdm-ds-0f601751-j488 61m 0% 6357Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 16756Mi 15% gke-xlou-cdm-ds-0f601751-nj6q 89m 0% 7047Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 57m 0% 12671Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 69m 0% 10742Mi 18% 16:08:11 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-848586d8db-b48f9 9m 2828Mi am-848586d8db-f4ck2 5m 1075Mi am-848586d8db-hl5g5 8m 2582Mi ds-cts-0 5m 11997Mi ds-cts-1 4m 11891Mi ds-cts-2 4m 10229Mi ds-idrepo-0 8m 3373Mi ds-idrepo-1 9m 3276Mi ds-idrepo-2 6m 3197Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 3m 1217Mi lodemon-69554696-dct8l 2m 55Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1m 6937Mi 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG { 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 DEBUG "data": { 16:08:11 DEBUG { 16:08:11 DEBUG { 16:08:11 DEBUG "data": { 16:08:11 DEBUG { 16:08:11 DEBUG "result": [ 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG "result": [ 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG "resultType": "vector", 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG --- stderr --- 16:08:11 DEBUG --- stderr --- 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG "result": [ 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG "resultType": "vector", 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 INFO Thread started 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG { 16:08:11 DEBUG "data": { 16:08:11 DEBUG "result": [ 16:08:11 DEBUG "result": [ 16:08:11 DEBUG { 16:08:11 DEBUG { 16:08:11 DEBUG "result": [ 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG "data": { 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG { 16:08:11 DEBUG { 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 16:08:11 DEBUG "data": { 16:08:11 DEBUG "result": [ 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713539291" 16:08:11 DEBUG "data": { 16:08:11 INFO Thread started 16:08:11 DEBUG "data": { 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539291" 16:08:11 DEBUG "data": { 16:08:11 DEBUG "metric": { 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG { 16:08:11 DEBUG { 16:08:11 DEBUG "metric": { 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG { 16:08:11 DEBUG "metric": { 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG "metric": { 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG { 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG "result": [ 16:08:11 DEBUG "resultType": "vector", 16:08:11 INFO Thread started 16:08:11 DEBUG "resultType": "vector", 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:08:11 DEBUG "result": [ 16:08:11 DEBUG "metric": { 16:08:11 DEBUG "metric": { 16:08:11 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:08:11 DEBUG "data": { 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG "data": { 16:08:11 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:08:11 DEBUG }, 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:08:11 DEBUG "metric": { 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG "result": [ 16:08:11 DEBUG { 16:08:11 DEBUG "result": [ 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539291" 16:08:11 DEBUG "result": [ 16:08:11 INFO Thread started 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG }, 16:08:11 DEBUG { 16:08:11 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:08:11 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:08:11 DEBUG }, 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG "result": [ 16:08:11 DEBUG }, 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:08:11 DEBUG "result": [ 16:08:11 DEBUG "value": [ 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG "instance": "10.142.0.107:9100" 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG { 16:08:11 DEBUG "metric": { 16:08:11 DEBUG { 16:08:11 DEBUG { 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539291" 16:08:11 DEBUG "value": [ 16:08:11 INFO Thread started 16:08:11 DEBUG "metric": { 16:08:11 DEBUG }, 16:08:11 DEBUG }, 16:08:11 DEBUG "value": [ 16:08:11 DEBUG "result": [ 16:08:11 DEBUG { 16:08:11 ERROR [http_cmd]: ERROR 16:08:11 DEBUG "value": [ 16:08:11 DEBUG "result": [ 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG { 16:08:11 DEBUG 16:08:11 DEBUG { 16:08:11 DEBUG }, 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:08:11 DEBUG "metric": { 16:08:11 DEBUG "instance": "10.142.0.107:9100" 16:08:11 DEBUG "metric": { 16:08:11 DEBUG "metric": { 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG 16:08:11 INFO All threads has been started 127.0.0.1 - - [19/Apr/2024 16:08:11] "GET /monitoring/start HTTP/1.1" 200 - 16:08:11 DEBUG "pod": "am-848586d8db-b48f9" 16:08:11 DEBUG "value": [ 16:08:11 DEBUG "value": [ 16:08:11 DEBUG 16:08:11 DEBUG { 16:08:11 DEBUG "metric": { 16:08:11 ERROR --- status code --- 16:08:11 DEBUG 16:08:11 DEBUG { 16:08:11 DEBUG { 16:08:11 DEBUG "metric": { 16:08:11 DEBUG 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG "value": [ 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG "instance": "10.142.0.107:9100" 16:08:11 DEBUG }, 16:08:11 DEBUG "pod": "ds-cts-0" 16:08:11 DEBUG "pod": "am-848586d8db-b48f9" 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:08:11 DEBUG 16:08:11 DEBUG }, 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "metric": { 16:08:11 DEBUG "pod": "ds-idrepo-0" 16:08:11 ERROR http status code is 400 (expected [200, 401, 403, 404]) 16:08:11 DEBUG 16:08:11 DEBUG "metric": { 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG "pod": "ds-idrepo-0" 16:08:11 DEBUG 16:08:11 DEBUG "data": { 16:08:11 DEBUG 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:08:11 DEBUG { 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG { 16:08:11 DEBUG }, 16:08:11 DEBUG "value": [ 16:08:11 DEBUG }, 16:08:11 DEBUG }, 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG 16:08:11 DEBUG "value": [ 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "pod": "ds-idrepo-0" 16:08:11 DEBUG }, 16:08:11 ERROR --- http response --- 16:08:11 DEBUG 16:08:11 DEBUG "pod": "am-848586d8db-b48f9" 16:08:11 DEBUG "data": { 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG [...] 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG "value": [ 16:08:11 DEBUG 16:08:11 DEBUG "value": [ 16:08:11 DEBUG 16:08:11 DEBUG { 16:08:11 DEBUG [...] 16:08:11 DEBUG 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG [...] 16:08:11 DEBUG }, 16:08:11 DEBUG "value": [ 16:08:11 DEBUG [...] 16:08:11 DEBUG }, 16:08:11 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 16:08:11 DEBUG }, 16:08:11 DEBUG "result": [ 16:08:11 DEBUG 16:08:11 DEBUG "result": [ 16:08:11 DEBUG 16:08:11 DEBUG { 16:08:11 DEBUG "data": { 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:08:11 DEBUG "data": { 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "value": [ 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG [...] 16:08:11 DEBUG [...] 16:08:11 DEBUG 16:08:11 DEBUG "value": [ 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "value": [ 16:08:11 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 16:08:11 DEBUG "value": [ 16:08:11 DEBUG { 16:08:11 DEBUG 16:08:11 DEBUG { 16:08:11 DEBUG [...] 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "data": { 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "metric": { 16:08:11 DEBUG 16:08:11 DEBUG "metric": { 16:08:11 DEBUG 16:08:11 DEBUG "data": { 16:08:11 DEBUG "result": [ 16:08:11 DEBUG { 16:08:11 DEBUG "result": [ 16:08:11 DEBUG 16:08:11 DEBUG [...] 16:08:11 DEBUG [...] 16:08:11 DEBUG 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG 16:08:11 DEBUG [...] 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 INFO [http_cmd]: http status code OK 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "pod": "ds-cts-0" 16:08:11 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 16:08:11 DEBUG "pod": "am-848586d8db-b48f9" 16:08:11 DEBUG 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG { 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG { 16:08:11 DEBUG [...] 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "result": [ 16:08:11 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:08:11 DEBUG --- status code --- 16:08:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:08:11 DEBUG [...] 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG }, 16:08:11 DEBUG "value": [ 16:08:11 DEBUG }, 16:08:11 DEBUG 16:08:11 DEBUG "result": [ 16:08:11 DEBUG "metric": { 16:08:11 DEBUG "data": { 16:08:11 DEBUG "metric": { 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG [...] 16:08:11 DEBUG { 16:08:11 DEBUG }, 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "pod": "idm-b85f9d978-wnhxb" 16:08:11 DEBUG }, 16:08:11 DEBUG 16:08:11 DEBUG --- http response --- 16:08:11 DEBUG "pod": "idm-b85f9d978-wnhxb" 16:08:11 DEBUG 16:08:11 DEBUG [...] 16:08:11 DEBUG [...] 16:08:11 DEBUG }, 16:08:11 DEBUG 16:08:11 DEBUG "value": [ 16:08:11 DEBUG "instance": "10.142.0.43:9100" 16:08:11 DEBUG { 16:08:11 DEBUG "pod": "am-848586d8db-b48f9" 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG "pod": "ds-cts-0" 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "metric": { 16:08:11 DEBUG "value": [ 16:08:11 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:08:11 DEBUG "pod": "am-848586d8db-f4ck2" 16:08:11 DEBUG }, 16:08:11 DEBUG "value": [ 16:08:11 DEBUG [...] 16:08:11 DEBUG }, 16:08:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "value": [ 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG }, 16:08:11 DEBUG "metric": { 16:08:11 DEBUG }, 16:08:11 DEBUG "result": [ 16:08:11 DEBUG }, 16:08:11 DEBUG 16:08:11 DEBUG "instance": "10.142.0.43:9100" 16:08:11 DEBUG "pod": "ds-cts-2" 16:08:11 DEBUG 16:08:11 DEBUG "pod": "am-848586d8db-b48f9" 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG }, 16:08:11 DEBUG }, 16:08:11 DEBUG "value": [ 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG 16:08:11 DEBUG "value": [ 16:08:11 DEBUG ----- output ----- 16:08:11 DEBUG "pod": "ds-idrepo-2" 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "value": [ 16:08:11 DEBUG "pod": "ds-idrepo-0" 16:08:11 DEBUG "value": [ 16:08:11 DEBUG { 16:08:11 DEBUG "value": [ 16:08:11 DEBUG "instance": "10.142.0.43:9100" 16:08:11 DEBUG }, 16:08:11 DEBUG }, 16:08:11 DEBUG 16:08:11 DEBUG }, 16:08:11 DEBUG "0" 16:08:11 DEBUG "value": [ 16:08:11 DEBUG "value": [ 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG "0.027446245973177526" 16:08:11 DEBUG 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG { 16:08:11 DEBUG }, 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "0" 16:08:11 DEBUG [...] 16:08:11 DEBUG 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG }, 16:08:11 DEBUG 16:08:11 DEBUG "metric": { 16:08:11 DEBUG 16:08:11 DEBUG }, 16:08:11 DEBUG "value": [ 16:08:11 DEBUG "value": [ 16:08:11 DEBUG "pod": "am-848586d8db-f4ck2" 16:08:11 DEBUG "value": [ 16:08:11 DEBUG ] 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG "0" 16:08:11 DEBUG ] 16:08:11 DEBUG 16:08:11 DEBUG "0" 16:08:11 DEBUG "status": "success", 16:08:11 DEBUG "data": { 16:08:11 DEBUG "pod": "am-848586d8db-f4ck2" 16:08:11 DEBUG "pod": "ds-idrepo-2" 16:08:11 DEBUG ] 16:08:11 DEBUG 16:08:11 DEBUG [...] 16:08:11 DEBUG "0.0009999999999990906" 16:08:11 DEBUG ] 16:08:11 DEBUG } 16:08:11 DEBUG "instance": "10.142.0.107:9100" 16:08:11 DEBUG 16:08:11 DEBUG "value": [ 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG }, 16:08:11 DEBUG 16:08:11 DEBUG } 16:08:11 DEBUG "7371186176" 16:08:11 DEBUG "0" 16:08:11 DEBUG ] 16:08:11 DEBUG } 16:08:11 DEBUG "pod": "ds-idrepo-2" 16:08:11 DEBUG ] 16:08:11 DEBUG "value": [ 16:08:11 DEBUG "resultType": "vector", 16:08:11 DEBUG }, 16:08:11 DEBUG }, 16:08:11 DEBUG } 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG "value": [ 16:08:11 DEBUG 16:08:11 DEBUG ] 16:08:11 DEBUG }, 16:08:11 DEBUG 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG "13952.833333333334" 16:08:11 DEBUG "0" 16:08:11 DEBUG "value": [ 16:08:11 DEBUG 16:08:11 DEBUG ] 16:08:11 DEBUG ] 16:08:11 DEBUG ] 16:08:11 DEBUG } 16:08:11 DEBUG ] 16:08:11 DEBUG }, 16:08:11 DEBUG } 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG "result": [ 16:08:11 DEBUG "value": [ 16:08:11 DEBUG "value": [ 16:08:11 DEBUG ] 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG } 16:08:11 DEBUG "value": [ 16:08:11 DEBUG [...] 16:08:11 DEBUG "14708.9" 16:08:11 DEBUG ] 16:08:11 DEBUG ] 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG ] 16:08:11 DEBUG } 16:08:11 DEBUG "value": [ 16:08:11 DEBUG ] 16:08:11 DEBUG "0" 16:08:11 DEBUG { 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG } 16:08:11 DEBUG "pod": "ds-cts-2" 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG [...] 16:08:11 DEBUG } 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG ] 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG "0" 16:08:11 DEBUG [...] 16:08:11 DEBUG } 16:08:11 DEBUG ] 16:08:11 DEBUG ] 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG } 16:08:11 DEBUG ] 16:08:11 DEBUG "metric": { 16:08:11 DEBUG "0" 16:08:11 DEBUG "0" 16:08:11 DEBUG } 16:08:11 DEBUG }, 16:08:11 DEBUG "pod": "am-848586d8db-f4ck2" 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG } 16:08:11 DEBUG ] 16:08:11 DEBUG ] 16:08:11 DEBUG ] 16:08:11 DEBUG 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG "0" 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG "instance": "10.142.0.107:9100" 16:08:11 DEBUG ] 16:08:11 DEBUG ] 16:08:11 DEBUG "value": [ 16:08:11 DEBUG }, 16:08:11 DEBUG [...] 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG ] 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG ] 16:08:11 DEBUG ] 16:08:11 DEBUG }, 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG "value": [ 16:08:11 DEBUG 16:08:11 DEBUG 16:08:11 DEBUG [...] 16:08:11 DEBUG "pod": "ds-cts-2" 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG ] 16:08:11 DEBUG 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG "value": [ 16:08:11 DEBUG ] 16:08:11 DEBUG ] 16:08:11 DEBUG "0" 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG "0" 16:08:11 DEBUG "pod": "am-848586d8db-f4ck2" 16:08:11 DEBUG 16:08:11 DEBUG }, 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG "pod": "am-848586d8db-f4ck2" 16:08:11 DEBUG ] 16:08:11 DEBUG } 16:08:11 DEBUG 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG ] 16:08:11 DEBUG 16:08:11 DEBUG ] 16:08:11 DEBUG }, 16:08:11 DEBUG 16:08:11 DEBUG "value": [ 16:08:11 DEBUG } 16:08:11 DEBUG }, 16:08:11 DEBUG } 16:08:11 DEBUG 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG 16:08:11 DEBUG } 16:08:11 DEBUG "value": [ 16:08:11 DEBUG 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG "value": [ 16:08:11 DEBUG } 16:08:11 DEBUG 16:08:11 DEBUG ] 16:08:11 DEBUG "pod": "ds-cts-2" 16:08:11 DEBUG ] 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG "instance": "10.142.0.43:9100" 16:08:11 DEBUG "0" 16:08:11 DEBUG ] 16:08:11 DEBUG [...] 16:08:11 DEBUG } 16:08:11 DEBUG }, 16:08:11 DEBUG } 16:08:11 DEBUG "1" 16:08:11 DEBUG }, 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG } 16:08:11 DEBUG 16:08:11 DEBUG } 16:08:11 DEBUG "value": [ 16:08:11 DEBUG } 16:08:11 DEBUG ] 16:08:11 DEBUG "value": [ 16:08:11 DEBUG "0" 16:08:11 DEBUG ] 16:08:11 DEBUG 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG } 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG ] 16:08:11 DEBUG } 16:08:11 DEBUG 16:08:11 DEBUG "0" 16:08:11 DEBUG ] 16:08:11 DEBUG "1609332204544" 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG "instance": "10.142.0.43:9100" 16:08:11 DEBUG ] 16:08:11 DEBUG } 16:08:11 DEBUG ] 16:08:11 DEBUG ] 16:08:11 DEBUG }, 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG "value": [ 16:08:11 DEBUG ] 16:08:11 DEBUG ] 16:08:11 DEBUG } 16:08:11 DEBUG 1713539291, 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG "0.0015666666667821499" 16:08:11 DEBUG } 16:08:11 DEBUG } 16:08:11 DEBUG ] 16:08:11 DEBUG } 16:08:11 DEBUG ] 16:08:11 DEBUG } 16:08:11 DEBUG } 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO 16:09:11 INFO [loop_until]: kubectl --namespace=xlou top pods 16:09:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG { 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "data": { 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "result": [ 16:09:11 DEBUG { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "pod": "am-848586d8db-b48f9" 16:09:11 DEBUG }, 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG [...] 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "pod": "am-848586d8db-f4ck2" 16:09:11 DEBUG }, 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG "0" 16:09:11 DEBUG ] 16:09:11 DEBUG } 16:09:11 DEBUG ] 16:09:11 DEBUG } 16:09:11 DEBUG } 16:09:11 INFO 16:09:11 INFO [loop_until]: kubectl --namespace=xlou top node 16:09:11 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 ERROR [http_cmd]: ERROR 16:09:11 ERROR --- status code --- 16:09:11 ERROR http status code is 400 (expected [200, 401, 403, 404]) 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 ERROR --- http response --- 16:09:11 DEBUG --- status code --- 16:09:11 INFO [loop_until]: OK (rc = 0) 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539351" 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713539351" 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 DEBUG --- stdout --- 16:09:11 DEBUG --- http response --- 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG --- status code --- 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 DEBUG --- status code --- 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-848586d8db-b48f9 1881m 3303Mi am-848586d8db-f4ck2 3393m 2610Mi am-848586d8db-hl5g5 3178m 3575Mi ds-cts-0 916m 12006Mi ds-cts-1 547m 11901Mi ds-cts-2 1161m 10274Mi ds-idrepo-0 796m 3373Mi ds-idrepo-1 751m 3278Mi ds-idrepo-2 531m 3197Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-69554696-dct8l 2m 61Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 3001m 7307Mi 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713539351" 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 DEBUG --- stderr --- 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG --- status code --- 16:09:11 INFO [loop_until]: OK (rc = 0) 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG --- stdout --- 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG { 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:09:11 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6198Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1169m 3% 6939Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2588m 8% 6113Mi 5% gke-xlou-cdm-default-pool-1b0769c8-qkns 2779m 9% 6617Mi 5% gke-xlou-cdm-default-pool-1b0769c8-w15q 1428m 4% 14998Mi 13% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 5199Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 48m 0% 2757Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 5039m 16% 6657Mi 5% gke-xlou-cdm-ds-0f601751-hmm7 50m 0% 2169Mi 1% gke-xlou-cdm-ds-0f601751-j488 564m 1% 6408Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 698m 2% 16570Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 1266m 4% 7077Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 1405m 4% 12696Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 2769m 17% 11095Mi 18% 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:09:11 DEBUG --- stderr --- 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG "data": { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:09:11 DEBUG ----- output ----- 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 DEBUG ----- output ----- 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 DEBUG 16:09:11 DEBUG ----- output ----- 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 INFO [http_cmd]: http status code OK 16:09:11 DEBUG { 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG { 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG { 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG { 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG --- status code --- 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "data": { 16:09:11 DEBUG "data": { 16:09:11 DEBUG "data": { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "data": { 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG "data": { 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG "data": { 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG --- http response --- 16:09:11 DEBUG { 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "data": { 16:09:11 DEBUG "data": { 16:09:11 DEBUG "data": { 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:09:11 DEBUG "metric": { 16:09:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "result": [ 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG "result": [ 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG "result": [ 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:09:11 DEBUG ----- output ----- 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG { 16:09:11 DEBUG "instance": "10.142.0.107:9100" 16:09:11 DEBUG "pod": "am-848586d8db-b48f9" 16:09:11 DEBUG "instance": "10.142.0.107:9100" 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG }, 16:09:11 DEBUG "data": { 16:09:11 DEBUG "status": "success", 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:09:11 DEBUG "data": { 16:09:11 DEBUG "data": { 16:09:11 DEBUG "data": { 16:09:11 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:09:11 DEBUG "data": { 16:09:11 DEBUG "data": { 16:09:11 DEBUG "data": { 16:09:11 DEBUG "data": { 16:09:11 DEBUG "data": { 16:09:11 DEBUG "data": { 16:09:11 DEBUG "data": { 16:09:11 DEBUG "data": { 16:09:11 DEBUG "instance": "10.142.0.107:9100" 16:09:11 DEBUG "data": { 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "data": { 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "instance": "10.142.0.107:9100" 16:09:11 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:09:11 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:09:11 DEBUG }, 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG }, 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG }, 16:09:11 DEBUG "resultType": "vector", 16:09:11 DEBUG 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "result": [ 16:09:11 DEBUG 16:09:11 DEBUG { 16:09:11 DEBUG 16:09:11 DEBUG { 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG { 16:09:11 DEBUG 16:09:11 DEBUG { 16:09:11 DEBUG 16:09:11 DEBUG "metric": { 16:09:11 DEBUG 16:09:11 DEBUG "metric": { 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG "metric": { 16:09:11 DEBUG 16:09:11 DEBUG "metric": { 16:09:11 DEBUG [...] 16:09:11 DEBUG "pod": "ds-cts-0" 16:09:11 DEBUG 16:09:11 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:09:11 DEBUG [...] 16:09:11 DEBUG [...] 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "pod": "ds-cts-0" 16:09:11 DEBUG "pod": "ds-idrepo-0" 16:09:11 DEBUG "pod": "ds-cts-0" 16:09:11 DEBUG 16:09:11 DEBUG "pod": "am-848586d8db-b48f9" 16:09:11 DEBUG "pod": "ds-idrepo-0" 16:09:11 DEBUG "pod": "ds-idrepo-0" 16:09:11 DEBUG "pod": "am-848586d8db-b48f9" 16:09:11 DEBUG "pod": "am-848586d8db-b48f9" 16:09:11 DEBUG "pod": "am-848586d8db-b48f9" 16:09:11 DEBUG "instance": "10.142.0.107:9100" 16:09:11 DEBUG "pod": "am-848586d8db-b48f9" 16:09:11 DEBUG 16:09:11 DEBUG "pod": "ds-idrepo-0" 16:09:11 DEBUG 16:09:11 DEBUG }, 16:09:11 DEBUG [...] 16:09:11 DEBUG }, 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG [...] 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG [...] 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG [...] 16:09:11 DEBUG }, 16:09:11 DEBUG 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG [...] 16:09:11 DEBUG [...] 16:09:11 DEBUG [...] 16:09:11 DEBUG 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:09:11 DEBUG [...] 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "pod": "am-848586d8db-f4ck2" 16:09:11 DEBUG "instance": "10.142.0.43:9100" 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "pod": "ds-idrepo-0" 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "pod": "ds-idrepo-0" 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG [...] 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "instance": "10.142.0.43:9100" 16:09:11 DEBUG }, 16:09:11 DEBUG [...] 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "instance": "10.142.0.43:9100" 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "instance": "10.142.0.43:9100" 16:09:11 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:09:11 DEBUG "pod": "ds-idrepo-0" 16:09:11 DEBUG }, 16:09:11 DEBUG [...] 16:09:11 DEBUG [...] 16:09:11 DEBUG [...] 16:09:11 DEBUG }, 16:09:11 DEBUG [...] 16:09:11 DEBUG 16:09:11 DEBUG [...] 16:09:11 DEBUG 16:09:11 DEBUG [...] 16:09:11 DEBUG [...] 16:09:11 DEBUG [...] 16:09:11 DEBUG [...] 16:09:11 DEBUG }, 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 16:09:11 DEBUG "pod": "ds-cts-2" 16:09:11 DEBUG 16:09:11 DEBUG }, 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 16:09:11 DEBUG [...] 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG 16:09:11 DEBUG }, 16:09:11 DEBUG [...] 16:09:11 DEBUG 16:09:11 DEBUG "0" 16:09:11 DEBUG "0.03353333333333334" 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "pod": "ds-idrepo-2" 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG "7717371904" 16:09:11 DEBUG 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 16:09:11 DEBUG ] 16:09:11 DEBUG ] 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG "0" 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "0.15566151495547634" 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG }, 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG 16:09:11 DEBUG "34297490.733333334" 16:09:11 DEBUG ] 16:09:11 DEBUG "pod": "ds-cts-2" 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG 16:09:11 DEBUG } 16:09:11 DEBUG } 16:09:11 DEBUG "0.02366666666666977" 16:09:11 DEBUG "0.812575931756469" 16:09:11 DEBUG "3159.483038302807" 16:09:11 DEBUG ] 16:09:11 DEBUG "pod": "ds-cts-2" 16:09:11 DEBUG "pod": "ds-idrepo-2" 16:09:11 DEBUG "pod": "ds-cts-2" 16:09:11 DEBUG ] 16:09:11 DEBUG "pod": "am-848586d8db-f4ck2" 16:09:11 DEBUG 16:09:11 DEBUG "pod": "ds-idrepo-2" 16:09:11 DEBUG 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "pod": "am-848586d8db-f4ck2" 16:09:11 DEBUG "instance": "10.142.0.43:9100" 16:09:11 DEBUG "pod": "am-848586d8db-f4ck2" 16:09:11 DEBUG ] 16:09:11 DEBUG } 16:09:11 DEBUG }, 16:09:11 DEBUG "0" 16:09:11 DEBUG "1609710568448" 16:09:11 DEBUG "pod": "ds-idrepo-0" 16:09:11 DEBUG ] 16:09:11 DEBUG ] 16:09:11 DEBUG ] 16:09:11 DEBUG } 16:09:11 DEBUG ] 16:09:11 DEBUG } 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG } 16:09:11 DEBUG }, 16:09:11 DEBUG "pod": "am-848586d8db-f4ck2" 16:09:11 DEBUG }, 16:09:11 DEBUG "pod": "am-848586d8db-f4ck2" 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG }, 16:09:11 DEBUG } 16:09:11 DEBUG ] 16:09:11 DEBUG "value": [ 16:09:11 DEBUG ] 16:09:11 DEBUG ] 16:09:11 DEBUG }, 16:09:11 DEBUG } 16:09:11 DEBUG } 16:09:11 DEBUG ] 16:09:11 DEBUG ] 16:09:11 DEBUG } 16:09:11 DEBUG ] 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG ] 16:09:11 DEBUG "value": [ 16:09:11 DEBUG }, 16:09:11 DEBUG "value": [ 16:09:11 DEBUG }, 16:09:11 DEBUG "0" 16:09:11 DEBUG ] 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG ] 16:09:11 DEBUG } 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG } 16:09:11 DEBUG } 16:09:11 DEBUG "value": [ 16:09:11 DEBUG } 16:09:11 DEBUG } 16:09:11 DEBUG } 16:09:11 DEBUG } 16:09:11 DEBUG } 16:09:11 DEBUG } 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG } 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG "value": [ 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG "value": [ 16:09:11 DEBUG "value": [ 16:09:11 DEBUG } 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG } 16:09:11 DEBUG } 16:09:11 DEBUG "0" 16:09:11 DEBUG ] 16:09:11 DEBUG ] 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG ] 16:09:11 DEBUG ] 16:09:11 DEBUG } 16:09:11 DEBUG "0" 16:09:11 DEBUG ] 16:09:11 DEBUG "0.00010619555555555556" 16:09:11 DEBUG ] 16:09:11 DEBUG } 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG "0" 16:09:11 DEBUG 1713539351, 16:09:11 DEBUG 1713539351, 16:09:12 DEBUG "505.8088377777778" 16:09:11 DEBUG "34296798.766666666" 16:09:11 DEBUG "10830.5" 16:09:11 DEBUG } 16:09:11 DEBUG ] 16:09:11 DEBUG } 16:09:11 DEBUG } 16:09:11 DEBUG "36334.05494048228" 16:09:11 DEBUG } 16:09:12 DEBUG } 16:09:11 DEBUG "0" 16:09:11 DEBUG } 16:09:11 DEBUG "724.4666666666667" 16:09:11 DEBUG } 16:09:11 DEBUG "504.87741555555556" 16:09:11 DEBUG ] 16:09:12 DEBUG "0" 16:09:11 DEBUG ] 16:09:12 DEBUG ] 16:09:12 DEBUG ] 16:09:12 DEBUG ] 16:09:12 DEBUG } 16:09:12 DEBUG ] 16:09:12 DEBUG } 16:09:12 DEBUG ] 16:09:11 DEBUG } 16:09:12 DEBUG ] 16:09:12 DEBUG ] 16:09:12 DEBUG ] 16:09:12 DEBUG ] 16:09:12 DEBUG ] 16:09:12 DEBUG } 16:09:12 DEBUG ] 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG ] 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG ] 16:09:12 DEBUG ] 16:09:12 DEBUG ] 16:09:12 DEBUG } 16:09:12 DEBUG ] 16:09:12 DEBUG ] 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG ] 16:09:12 DEBUG } 16:09:12 DEBUG ] 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG ] 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:09:12 DEBUG } 16:10:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539411" 16:10:11 INFO [http_cmd]: http status code OK 16:10:11 DEBUG --- status code --- 16:10:11 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:11 DEBUG --- http response --- 16:10:11 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:10:11 DEBUG ----- output ----- 16:10:11 DEBUG { 16:10:11 DEBUG "status": "success", 16:10:11 DEBUG "data": { 16:10:11 DEBUG "resultType": "vector", 16:10:11 DEBUG "result": [ 16:10:11 DEBUG { 16:10:11 DEBUG "metric": { 16:10:11 DEBUG "pod": "am-848586d8db-b48f9" 16:10:11 DEBUG }, 16:10:11 DEBUG "value": [ 16:10:11 DEBUG 16:10:11 DEBUG 16:10:11 DEBUG 16:10:11 DEBUG [...] 16:10:11 DEBUG 16:10:11 DEBUG 16:10:11 DEBUG 16:10:11 DEBUG "pod": "am-848586d8db-f4ck2" 16:10:11 DEBUG }, 16:10:11 DEBUG "value": [ 16:10:11 DEBUG 1713539411, 16:10:11 DEBUG "0" 16:10:11 DEBUG ] 16:10:11 DEBUG } 16:10:11 DEBUG ] 16:10:11 DEBUG } 16:10:11 DEBUG } 16:10:11 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713539411" 16:10:12 ERROR [http_cmd]: ERROR 16:10:12 ERROR --- status code --- 16:10:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 16:10:12 ERROR --- http response --- 16:10:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 16:10:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO 16:10:12 INFO [loop_until]: kubectl --namespace=xlou top pods 16:10:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713539412" 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG --- http response --- 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:10:12 INFO [loop_until]: kubectl --namespace=xlou top node 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 DEBUG { 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG { 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG "data": { 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG --- http response --- 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 DEBUG --- http response --- 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 DEBUG "resultType": "vector", 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 DEBUG "data": { 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG "result": [ 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG ----- output ----- 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG { 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG "result": [ 16:10:12 DEBUG "pod": "ds-idrepo-0" 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:10:12 DEBUG { 16:10:12 DEBUG }, 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:10:12 DEBUG "data": { 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "value": [ 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG "pod": "ds-cts-0" 16:10:12 DEBUG 16:10:12 DEBUG { 16:10:12 DEBUG "result": [ 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG }, 16:10:12 DEBUG 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG "data": { 16:10:12 DEBUG "data": { 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 16:10:12 DEBUG "data": { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "result": [ 16:10:12 DEBUG [...] 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:10:12 DEBUG "data": { 16:10:12 DEBUG "result": [ 16:10:12 DEBUG 16:10:12 DEBUG { 16:10:12 DEBUG 16:10:12 DEBUG "result": [ 16:10:12 DEBUG }, 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG { 16:10:12 DEBUG 16:10:12 DEBUG "metric": { 16:10:12 DEBUG 16:10:12 DEBUG { 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "result": [ 16:10:12 DEBUG "metric": { 16:10:12 DEBUG 16:10:12 DEBUG "pod": "am-848586d8db-b48f9" 16:10:12 DEBUG 16:10:12 DEBUG "metric": { 16:10:12 DEBUG 16:10:12 DEBUG { 16:10:12 DEBUG "pod": "ds-idrepo-0" 16:10:12 DEBUG [...] 16:10:12 DEBUG }, 16:10:12 DEBUG "pod": "ds-cts-2" 16:10:12 DEBUG "pod": "am-848586d8db-b48f9" 16:10:12 DEBUG 16:10:12 DEBUG "metric": { 16:10:12 DEBUG }, 16:10:12 DEBUG 16:10:12 DEBUG "value": [ 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG 16:10:12 DEBUG "pod": "am-848586d8db-b48f9" 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "value": [ 16:10:12 DEBUG [...] 16:10:12 DEBUG }, 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 16:10:12 DEBUG "pod": "ds-cts-2" 16:10:12 DEBUG 16:10:12 DEBUG "0" 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG }, 16:10:12 DEBUG [...] 16:10:12 DEBUG ] 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG [...] 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 16:10:12 DEBUG } 16:10:12 DEBUG [...] 16:10:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG 16:10:12 DEBUG ] 16:10:12 DEBUG 16:10:12 DEBUG }, 16:10:12 DEBUG [...] 16:10:12 DEBUG 16:10:12 DEBUG "0" 16:10:12 DEBUG 16:10:12 DEBUG } 16:10:12 DEBUG 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG ] 16:10:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:10:12 DEBUG } 16:10:12 DEBUG 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG 16:10:12 DEBUG "pod": "ds-idrepo-2" 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:10:12 DEBUG "7.364000270489018" 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 DEBUG 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG "value": [ 16:10:12 DEBUG }, 16:10:12 DEBUG ] 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:10:12 DEBUG "value": [ 16:10:12 DEBUG } 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG "value": [ 16:10:12 DEBUG } 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG }, 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG } 16:10:12 DEBUG "106260" 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG ] 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "0" 16:10:12 DEBUG ] 16:10:12 DEBUG "1427.4666666666667" 16:10:12 DEBUG } 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG "1450.1333333333332" 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG { 16:10:12 DEBUG ] 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG "data": { 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG } 16:10:12 DEBUG "result": [ 16:10:12 DEBUG } 16:10:12 DEBUG { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:10:12 DEBUG }, 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG [...] 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG "pod": "lodemon-69554696-dct8l" 16:10:12 DEBUG }, 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG "0" 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539412" 16:10:12 INFO [loop_until]: OK (rc = 0) 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 DEBUG --- stdout --- 16:10:12 DEBUG --- status code --- 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 DEBUG --- status code --- 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG --- status code --- 16:10:12 INFO [loop_until]: OK (rc = 0) 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-848586d8db-b48f9 8255m 4630Mi am-848586d8db-f4ck2 9066m 5332Mi am-848586d8db-hl5g5 7903m 4672Mi ds-cts-0 3876m 11358Mi ds-cts-1 4598m 11313Mi ds-cts-2 4849m 10300Mi ds-idrepo-0 3866m 3381Mi ds-idrepo-1 3648m 3278Mi ds-idrepo-2 3825m 3199Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-69554696-dct8l 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8061m 7530Mi 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG --- status code --- 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG --- stdout --- 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG --- stderr --- 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6213Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3735m 12% 6936Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8146m 27% 7380Mi 6% gke-xlou-cdm-default-pool-1b0769c8-qkns 7996m 26% 7808Mi 6% gke-xlou-cdm-default-pool-1b0769c8-w15q 3810m 12% 14418Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 149m 0% 5180Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 51m 0% 2762Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 22296m 74% 9088Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 51m 0% 2170Mi 1% gke-xlou-cdm-ds-0f601751-j488 4108m 13% 6417Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 4655m 15% 16086Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3816m 12% 7090Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 4177m 13% 12790Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8138m 51% 11180Mi 19% 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:10:12 DEBUG --- stderr --- 16:10:12 INFO [http_cmd]: http status code OK 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG 16:10:12 DEBUG --- status code --- 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG { 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG --- http response --- 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG "data": { 16:10:12 DEBUG { 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG "data": { 16:10:12 DEBUG "data": { 16:10:12 DEBUG "data": { 16:10:12 DEBUG "data": { 16:10:12 DEBUG "data": { 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:10:12 DEBUG "data": { 16:10:12 DEBUG "data": { 16:10:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:10:12 DEBUG "data": { 16:10:12 DEBUG "data": { 16:10:12 DEBUG "data": { 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG "data": { 16:10:12 DEBUG "data": { 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG ----- output ----- 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "result": [ 16:10:12 DEBUG "data": { 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "result": [ 16:10:12 DEBUG "result": [ 16:10:12 DEBUG "result": [ 16:10:12 DEBUG "result": [ 16:10:12 DEBUG "result": [ 16:10:12 DEBUG { 16:10:12 DEBUG "result": [ 16:10:12 DEBUG "result": [ 16:10:12 DEBUG { 16:10:12 DEBUG "result": [ 16:10:12 DEBUG "result": [ 16:10:12 DEBUG "result": [ 16:10:12 DEBUG { 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "result": [ 16:10:12 DEBUG "result": [ 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG "status": "success", 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "result": [ 16:10:12 DEBUG { 16:10:12 DEBUG { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "data": { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "data": { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "pod": "am-848586d8db-b48f9" 16:10:12 DEBUG { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "instance": "10.142.0.107:9100" 16:10:12 DEBUG }, 16:10:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:10:12 DEBUG "instance": "10.142.0.107:9100" 16:10:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:10:12 DEBUG "instance": "10.142.0.107:9100" 16:10:12 DEBUG "resultType": "vector", 16:10:12 DEBUG "pod": "ds-cts-0" 16:10:12 DEBUG "pod": "ds-cts-0" 16:10:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:10:12 DEBUG }, 16:10:12 DEBUG "metric": { 16:10:12 DEBUG "pod": "ds-idrepo-0" 16:10:12 DEBUG "pod": "am-848586d8db-b48f9" 16:10:12 DEBUG "instance": "10.142.0.107:9100" 16:10:12 DEBUG "value": [ 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG "result": [ 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG "result": [ 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "instance": "10.142.0.107:9100" 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "value": [ 16:10:12 DEBUG { 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "value": [ 16:10:12 DEBUG { 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 16:10:12 DEBUG }, 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG "metric": { 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG "metric": { 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG "pod": "ds-idrepo-0" 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG "pod": "am-848586d8db-b48f9" 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG [...] 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG }, 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG }, 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG [...] 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG [...] 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG [...] 16:10:12 DEBUG [...] 16:10:12 DEBUG [...] 16:10:12 DEBUG "value": [ 16:10:12 DEBUG [...] 16:10:12 DEBUG [...] 16:10:12 DEBUG "value": [ 16:10:12 DEBUG [...] 16:10:12 DEBUG [...] 16:10:12 DEBUG [...] 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG [...] 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG [...] 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG "pod": "ds-idrepo-2" 16:10:12 DEBUG "instance": "10.142.0.43:9100" 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG 16:10:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:10:12 DEBUG 16:10:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:10:12 DEBUG [...] 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG "pod": "lodemon-69554696-dct8l" 16:10:12 DEBUG "instance": "10.142.0.43:9100" 16:10:12 DEBUG "pod": "lodemon-69554696-dct8l" 16:10:12 DEBUG [...] 16:10:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:10:12 DEBUG "instance": "10.142.0.43:9100" 16:10:12 DEBUG [...] 16:10:12 DEBUG "pod": "ds-cts-2" 16:10:12 DEBUG "pod": "ds-cts-2" 16:10:12 DEBUG "pod": "lodemon-69554696-dct8l" 16:10:12 DEBUG }, 16:10:12 DEBUG 16:10:12 DEBUG }, 16:10:12 DEBUG 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "value": [ 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG }, 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "instance": "10.142.0.43:9100" 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "value": [ 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG }, 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG 16:10:12 DEBUG "0" 16:10:12 DEBUG "0.03213333333340529" 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG "0" 16:10:12 DEBUG "value": [ 16:10:12 DEBUG "1446.6666666666667" 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG "0" 16:10:12 DEBUG "0.024000000000000906" 16:10:12 DEBUG "0.0003432173123501804" 16:10:12 DEBUG "pod": "ds-idrepo-2" 16:10:12 DEBUG "7878119424" 16:10:12 DEBUG "1611443010560" 16:10:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:10:12 DEBUG "0" 16:10:12 DEBUG "0" 16:10:12 DEBUG "2026072.7790613186" 16:10:12 DEBUG ] 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG "instance": "10.142.0.43:9100" 16:10:12 DEBUG }, 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG ] 16:10:12 DEBUG ] 16:10:12 DEBUG }, 16:10:12 DEBUG ] 16:10:12 DEBUG ] 16:10:12 DEBUG }, 16:10:12 DEBUG ] 16:10:12 DEBUG ] 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG "35455355.766666666" 16:10:12 DEBUG ] 16:10:12 DEBUG "value": [ 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG "value": [ 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG "value": [ 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG ] 16:10:12 DEBUG ] 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG ] 16:10:12 DEBUG ] 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG ] 16:10:12 DEBUG ] 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG 1713539412, 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG "0" 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG "0" 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG "35455991" 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG ] 16:10:12 DEBUG ] 16:10:12 DEBUG ] 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:10:12 DEBUG } 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713539472" 16:11:12 ERROR [http_cmd]: ERROR 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 ERROR --- status code --- 16:11:12 DEBUG --- status code --- 16:11:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 ERROR --- http response --- 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 16:11:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "data": { 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "result": [ 16:11:12 DEBUG { 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "pod": "am-848586d8db-b48f9" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG [...] 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG "0" 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "data": { 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "result": [ 16:11:12 DEBUG { 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "pod": "am-848586d8db-b48f9" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG [...] 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG "191392" 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713539472" 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "data": { 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "result": [ 16:11:12 DEBUG { 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG [...] 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG "8.036217831509461" 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- http response --- 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "data": { 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "result": [ 16:11:12 DEBUG { 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "pod": "ds-idrepo-0" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG [...] 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "pod": "ds-cts-2" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG "0" 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG { 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "data": { 16:11:12 DEBUG "status": "success", 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 DEBUG "data": { 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "data": { 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "result": [ 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "result": [ 16:11:12 DEBUG { 16:11:12 DEBUG "result": [ 16:11:12 DEBUG { 16:11:12 DEBUG "metric": { 16:11:12 DEBUG { 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "pod": "ds-idrepo-0" 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "pod": "am-848586d8db-b48f9" 16:11:12 DEBUG }, 16:11:12 DEBUG "pod": "ds-cts-0" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG [...] 16:11:12 DEBUG 16:11:12 DEBUG [...] 16:11:12 DEBUG 16:11:12 DEBUG [...] 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "pod": "ds-idrepo-2" 16:11:12 DEBUG 16:11:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:11:12 DEBUG }, 16:11:12 DEBUG "pod": "ds-cts-2" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG "0" 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG "1397.0666666666666" 16:11:12 DEBUG ] 16:11:12 DEBUG "0" 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 INFO 16:11:12 INFO [loop_until]: kubectl --namespace=xlou top node 16:11:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539472" 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "data": { 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "result": [ 16:11:12 DEBUG { 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG [...] 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "pod": "ds-cts-1" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG "105.75806517729043" 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "data": { 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "result": [ 16:11:12 DEBUG { 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "pod": "am-848586d8db-b48f9" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG [...] 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG "1395.6333333333332" 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "data": { 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "result": [ 16:11:12 DEBUG { 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "instance": "10.142.0.107:9100" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG [...] 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "instance": "10.142.0.43:9100" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG "1612931812352" 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 INFO 16:11:12 INFO [loop_until]: kubectl --namespace=xlou top pods 16:11:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "data": { 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "result": [ 16:11:12 DEBUG { 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG [...] 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "pod": "ds-cts-1" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG "1.472586995915369" 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- http response --- 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "data": { 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "result": [ 16:11:12 DEBUG { 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "pod": "am-848586d8db-b48f9" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG [...] 16:11:12 DEBUG 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG "1397.3333333333333" 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "data": { 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "result": [ 16:11:12 DEBUG { 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "pod": "am-848586d8db-b48f9" 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG [...] 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:11:12 DEBUG 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG { 16:11:12 DEBUG { 16:11:12 DEBUG 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG "data": { 16:11:12 DEBUG "data": { 16:11:12 DEBUG }, 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "value": [ 16:11:12 DEBUG "result": [ 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG "result": [ 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG { 16:11:12 DEBUG { 16:11:12 DEBUG { 16:11:12 DEBUG "0.0005" 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "metric": { 16:11:12 DEBUG ] 16:11:12 DEBUG "pod": "ds-cts-0" 16:11:12 DEBUG "data": { 16:11:12 DEBUG "pod": "ds-idrepo-0" 16:11:12 DEBUG } 16:11:12 DEBUG }, 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG }, 16:11:12 DEBUG ] 16:11:12 DEBUG "value": [ 16:11:12 DEBUG "result": [ 16:11:12 DEBUG "value": [ 16:11:12 DEBUG } 16:11:12 DEBUG 16:11:12 DEBUG { 16:11:12 DEBUG 16:11:12 DEBUG } 16:11:12 DEBUG 16:11:12 DEBUG "metric": { 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "pod": "ds-idrepo-0" 16:11:12 DEBUG 16:11:12 DEBUG [...] 16:11:12 DEBUG }, 16:11:12 DEBUG [...] 16:11:12 DEBUG 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "pod": "ds-cts-2" 16:11:12 DEBUG 16:11:12 DEBUG "pod": "ds-idrepo-2" 16:11:12 DEBUG }, 16:11:12 DEBUG [...] 16:11:12 DEBUG }, 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG "0" 16:11:12 DEBUG 16:11:12 DEBUG "0" 16:11:12 DEBUG ] 16:11:12 DEBUG "pod": "ds-idrepo-2" 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG }, 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG "value": [ 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 DEBUG "0" 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539472" 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539472" 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713539472" 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539472" 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539472" 16:11:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539472" 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG --- status code --- 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:11:12 DEBUG ----- output ----- 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG ----- output ----- 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG --- status code --- 16:11:12 INFO [loop_until]: OK (rc = 0) 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG { 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG { 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- stdout --- 16:11:12 DEBUG { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG "status": "success", 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6223Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3238m 10% 6948Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6503m 21% 7405Mi 6% gke-xlou-cdm-default-pool-1b0769c8-qkns 6777m 22% 7834Mi 7% gke-xlou-cdm-default-pool-1b0769c8-w15q 3400m 11% 14407Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 151m 0% 5201Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 52m 0% 2760Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 17789m 59% 9114Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 52m 0% 2174Mi 1% gke-xlou-cdm-ds-0f601751-j488 3321m 11% 6417Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 3738m 12% 16100Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3472m 11% 7084Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 3675m 12% 12810Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8198m 51% 11087Mi 18% 16:11:12 DEBUG "data": { 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG "data": { 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG "data": { 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG --- stderr --- 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG 16:11:12 DEBUG "result": [ 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG ----- output ----- 16:11:12 INFO [http_cmd]: http status code OK 16:11:12 DEBUG "result": [ 16:11:12 DEBUG ----- output ----- 16:11:12 INFO [loop_until]: OK (rc = 0) 16:11:12 DEBUG "result": [ 16:11:12 DEBUG { 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG { 16:11:12 DEBUG { 16:11:12 DEBUG { 16:11:12 DEBUG --- status code --- 16:11:12 DEBUG { 16:11:12 DEBUG { 16:11:12 DEBUG --- stdout --- 16:11:12 DEBUG { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "metric": { 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "metric": { 16:11:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-848586d8db-b48f9 6678m 4641Mi am-848586d8db-f4ck2 8172m 5337Mi am-848586d8db-hl5g5 6615m 4677Mi ds-cts-0 3438m 11361Mi ds-cts-1 3624m 11342Mi ds-cts-2 3716m 10321Mi ds-idrepo-0 3278m 3390Mi ds-idrepo-1 3286m 3277Mi ds-idrepo-2 3171m 3200Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1174Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-69554696-dct8l 8m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8013m 7642Mi 16:11:12 DEBUG "data": { 16:11:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG "data": { 16:11:12 DEBUG "data": { 16:11:12 DEBUG --- http response --- 16:11:12 DEBUG "instance": "10.142.0.107:9100" 16:11:12 DEBUG "data": { 16:11:12 DEBUG "instance": "10.142.0.107:9100" 16:11:12 DEBUG }, 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG }, 16:11:12 DEBUG { 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG }, 16:11:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "value": [ 16:11:12 DEBUG --- stderr --- 16:11:12 DEBUG "result": [ 16:11:12 DEBUG "value": [ 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "result": [ 16:11:12 DEBUG "result": [ 16:11:12 DEBUG "value": [ 16:11:12 DEBUG ----- output ----- 16:11:12 DEBUG "result": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG { 16:11:12 DEBUG 16:11:12 DEBUG "data": { 16:11:12 DEBUG { 16:11:12 DEBUG { 16:11:12 DEBUG 16:11:12 DEBUG { 16:11:12 DEBUG { 16:11:12 DEBUG 16:11:12 DEBUG "metric": { 16:11:12 DEBUG 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "metric": { 16:11:12 DEBUG 16:11:12 DEBUG "status": "success", 16:11:12 DEBUG "metric": { 16:11:12 DEBUG 16:11:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:11:12 DEBUG 16:11:12 DEBUG "result": [ 16:11:12 DEBUG "instance": "10.142.0.107:9100" 16:11:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:11:12 DEBUG 16:11:12 DEBUG "data": { 16:11:12 DEBUG "pod": "am-848586d8db-b48f9" 16:11:12 DEBUG [...] 16:11:12 DEBUG }, 16:11:12 DEBUG [...] 16:11:12 DEBUG { 16:11:12 DEBUG }, 16:11:12 DEBUG }, 16:11:12 DEBUG [...] 16:11:12 DEBUG "resultType": "vector", 16:11:12 DEBUG }, 16:11:12 DEBUG 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG "metric": { 16:11:12 DEBUG "value": [ 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG "result": [ 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "pod": "ds-cts-0" 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG { 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG }, 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "metric": { 16:11:12 DEBUG 16:11:12 DEBUG "instance": "10.142.0.43:9100" 16:11:12 DEBUG 16:11:12 DEBUG "pod": "ds-cts-1" 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "instance": "10.142.0.43:9100" 16:11:12 DEBUG "instance": "10.142.0.107:9100" 16:11:12 DEBUG 16:11:12 DEBUG }, 16:11:12 DEBUG [...] 16:11:12 DEBUG }, 16:11:12 DEBUG 16:11:12 DEBUG [...] 16:11:12 DEBUG [...] 16:11:12 DEBUG }, 16:11:12 DEBUG }, 16:11:12 DEBUG [...] 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "value": [ 16:11:12 DEBUG "value": [ 16:11:12 DEBUG 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "0.02333333333332916" 16:11:12 DEBUG 16:11:12 DEBUG "10623291.888993647" 16:11:12 DEBUG [...] 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG "34066930.13333333" 16:11:12 DEBUG 16:11:12 DEBUG 16:11:12 DEBUG ] 16:11:12 DEBUG "pod": "ds-cts-1" 16:11:12 DEBUG ] 16:11:12 DEBUG 16:11:12 DEBUG "instance": "10.142.0.43:9100" 16:11:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:11:12 DEBUG ] 16:11:12 DEBUG 16:11:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:11:12 DEBUG } 16:11:12 DEBUG }, 16:11:12 DEBUG } 16:11:12 DEBUG 16:11:12 DEBUG }, 16:11:12 DEBUG }, 16:11:12 DEBUG } 16:11:12 DEBUG [...] 16:11:12 DEBUG }, 16:11:12 DEBUG ] 16:11:12 DEBUG "value": [ 16:11:12 DEBUG ] 16:11:12 DEBUG 16:11:12 DEBUG "value": [ 16:11:12 DEBUG "value": [ 16:11:12 DEBUG ] 16:11:12 DEBUG 16:11:12 DEBUG "value": [ 16:11:12 DEBUG } 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG } 16:11:12 DEBUG "pod": "ds-cts-2" 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG } 16:11:12 DEBUG 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG } 16:11:12 DEBUG "201574.87222791556" 16:11:12 DEBUG } 16:11:12 DEBUG }, 16:11:12 DEBUG "0.03246666666667106" 16:11:12 DEBUG "8061693952" 16:11:12 DEBUG } 16:11:12 DEBUG 16:11:12 DEBUG "0" 16:11:12 DEBUG ] 16:11:12 DEBUG "value": [ 16:11:12 DEBUG ] 16:11:12 DEBUG ] 16:11:12 DEBUG "instance": "10.142.0.43:9100" 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 DEBUG }, 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG "0" 16:11:12 DEBUG ] 16:11:12 DEBUG ] 16:11:12 DEBUG "value": [ 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 DEBUG 1713539472, 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 DEBUG "34070057.5" 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 DEBUG } 16:11:12 DEBUG ] 16:11:12 DEBUG } 16:11:12 DEBUG } 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "pod": "am-848586d8db-b48f9" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "517.4164599999999" 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713539532" 16:12:12 ERROR [http_cmd]: ERROR 16:12:12 ERROR --- status code --- 16:12:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 16:12:12 ERROR --- http response --- 16:12:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 16:12:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG { 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "pod": "am-848586d8db-b48f9" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "262364.5" 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "pod": "ds-idrepo-0" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "pod": "ds-cts-2" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "0" 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "pod": "ds-idrepo-0" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "pod": "ds-idrepo-2" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "0" 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "pod": "ds-cts-0" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "pod": "ds-cts-2" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "0" 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "8.03085822071402" 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "0.002989424355646695" 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "pod": "am-848586d8db-b48f9" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "1377.9333333333334" 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "pod": "ds-idrepo-0" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "pod": "ds-idrepo-2" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "0" 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539532" 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG { 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "pod": "am-848586d8db-b48f9" 16:12:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:12:12 DEBUG }, 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:12:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 16:12:12 DEBUG }, 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "1373.8" 16:12:12 DEBUG "0" 16:12:12 DEBUG ] 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "instance": "10.142.0.107:9100" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "instance": "10.142.0.43:9100" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "1614489900032" 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [loop_until]: kubectl --namespace=xlou top pods 16:12:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "pod": "am-848586d8db-b48f9" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "1376.1" 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG { 16:12:12 DEBUG { 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "data": { 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG "result": [ 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG { 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:12:12 DEBUG "pod": "ds-cts-0" 16:12:12 DEBUG "pod": "am-848586d8db-b48f9" 16:12:12 DEBUG }, 16:12:12 DEBUG }, 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG "value": [ 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG [...] 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 16:12:12 DEBUG "pod": "ds-cts-2" 16:12:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:12:12 DEBUG }, 16:12:12 DEBUG }, 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG "value": [ 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "907.8589069723262" 16:12:12 DEBUG "0" 16:12:12 DEBUG "0.001" 16:12:12 DEBUG ] 16:12:12 DEBUG ] 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG ] 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "pod": "ds-idrepo-0" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "pod": "ds-idrepo-2" 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "0" 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 INFO 16:12:12 INFO [loop_until]: kubectl --namespace=xlou top node 16:12:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539532" 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:12:12 DEBUG --- status code --- 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- status code --- 16:12:12 INFO [loop_until]: OK (rc = 0) 16:12:12 DEBUG --- status code --- 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG { 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- stdout --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-848586d8db-b48f9 7246m 7077Mi am-848586d8db-f4ck2 8830m 6875Mi am-848586d8db-hl5g5 7369m 7035Mi ds-cts-0 4589m 11365Mi ds-cts-1 3943m 11345Mi ds-cts-2 4210m 10345Mi ds-idrepo-0 3389m 3393Mi ds-idrepo-1 3458m 3278Mi ds-idrepo-2 3355m 3200Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-69554696-dct8l 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7930m 7870Mi 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG "data": { 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 INFO [http_cmd]: http status code OK 16:12:12 DEBUG --- stderr --- 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG { 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG --- status code --- 16:12:12 DEBUG 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG { 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG "result": [ 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG { 16:12:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG { 16:12:12 DEBUG "data": { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG --- http response --- 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "data": { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "data": { 16:12:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:12:12 DEBUG { 16:12:12 DEBUG "data": { 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "instance": "10.142.0.107:9100" 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG "result": [ 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG ----- output ----- 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG "result": [ 16:12:12 DEBUG }, 16:12:12 DEBUG { 16:12:12 DEBUG { 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG "data": { 16:12:12 DEBUG "result": [ 16:12:12 DEBUG { 16:12:12 DEBUG "value": [ 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "metric": { 16:12:12 DEBUG { 16:12:12 DEBUG "status": "success", 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG { 16:12:12 DEBUG "metric": { 16:12:12 DEBUG 16:12:12 DEBUG "data": { 16:12:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "data": { 16:12:12 DEBUG "result": [ 16:12:12 DEBUG "metric": { 16:12:12 DEBUG "instance": "10.142.0.107:9100" 16:12:12 DEBUG 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG }, 16:12:12 DEBUG "instance": "10.142.0.107:9100" 16:12:12 DEBUG "resultType": "vector", 16:12:12 DEBUG { 16:12:12 DEBUG "instance": "10.142.0.107:9100" 16:12:12 DEBUG }, 16:12:12 DEBUG 16:12:12 DEBUG "result": [ 16:12:12 DEBUG "value": [ 16:12:12 DEBUG }, 16:12:12 DEBUG "result": [ 16:12:12 DEBUG "metric": { 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG [...] 16:12:12 DEBUG { 16:12:12 DEBUG 16:12:12 DEBUG "value": [ 16:12:12 DEBUG { 16:12:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "metric": { 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "metric": { 16:12:12 DEBUG }, 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "pod": "am-848586d8db-b48f9" 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "pod": "ds-cts-0" 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG }, 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG }, 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG "instance": "10.142.0.43:9100" 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG }, 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG 16:12:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG "instance": "10.142.0.43:9100" 16:12:12 DEBUG "43892135.56666666" 16:12:12 DEBUG [...] 16:12:12 DEBUG }, 16:12:12 DEBUG "instance": "10.142.0.43:9100" 16:12:12 DEBUG [...] 16:12:12 DEBUG 16:12:12 DEBUG "instance": "10.142.0.43:9100" 16:12:12 DEBUG }, 16:12:12 DEBUG ] 16:12:12 DEBUG 16:12:12 DEBUG "value": [ 16:12:12 DEBUG }, 16:12:12 DEBUG 16:12:12 DEBUG 16:12:12 DEBUG }, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG } 16:12:12 DEBUG 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 16:12:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 16:12:12 DEBUG "value": [ 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG ] 16:12:12 DEBUG 16:12:12 DEBUG "8227782656" 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG 16:12:12 DEBUG }, 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG "0.02566666666666795" 16:12:12 DEBUG } 16:12:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:12:12 DEBUG ] 16:12:12 DEBUG "43892341.733333334" 16:12:12 DEBUG "pod": "ds-cts-2" 16:12:12 DEBUG "value": [ 16:12:12 DEBUG "0.0378333333333264" 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG }, 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG }, 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG "value": [ 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG "value": [ 16:12:12 DEBUG "20751.060730796024" 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG 1713539532, 16:12:12 DEBUG ] 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG "0" 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG "0.0002292933333333333" 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG ] 16:12:12 DEBUG ] 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 DEBUG } 16:12:12 INFO [loop_until]: OK (rc = 0) 16:12:12 DEBUG --- stdout --- 16:12:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6209Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3453m 11% 6948Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7350m 24% 9738Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 7342m 24% 10278Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 4926m 16% 14385Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5199Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 52m 0% 2753Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 19626m 65% 10657Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 52m 0% 2177Mi 1% gke-xlou-cdm-ds-0f601751-j488 3443m 11% 6419Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 4539m 15% 16108Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3621m 12% 7094Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 5397m 18% 12815Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8303m 52% 11089Mi 18% 16:12:12 DEBUG --- stderr --- 16:12:12 DEBUG 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "pod": "am-848586d8db-b48f9" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "1458.2333333333333" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713539592" 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "pod": "am-848586d8db-b48f9" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "247437.5" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 ERROR [http_cmd]: ERROR 16:13:12 ERROR --- status code --- 16:13:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 16:13:12 ERROR --- http response --- 16:13:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 16:13:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "pod": "ds-cts-0" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "pod": "ds-cts-2" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "0" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "pod": "ds-idrepo-0" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "pod": "ds-cts-2" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "0" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713539592" 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "pod": "ds-idrepo-0" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "pod": "ds-idrepo-2" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "0" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "8.029558935199251" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "pod": "am-848586d8db-hl5g5" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "8.30817692119631" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG { 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "data": { 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG "result": [ 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG { 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "pod": "am-848586d8db-b48f9" 16:13:12 DEBUG "pod": "ds-idrepo-0" 16:13:12 DEBUG "pod": "am-848586d8db-b48f9" 16:13:12 DEBUG }, 16:13:12 DEBUG }, 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG "value": [ 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG [...] 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:13:12 DEBUG "pod": "ds-idrepo-2" 16:13:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:13:12 DEBUG }, 16:13:12 DEBUG }, 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG "value": [ 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "0.0015" 16:13:12 DEBUG "0" 16:13:12 DEBUG "1287.9333333333334" 16:13:12 DEBUG ] 16:13:12 DEBUG ] 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG ] 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539592" 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG "result": [ 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:13:12 DEBUG { 16:13:12 DEBUG { 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG { 16:13:12 DEBUG "pod": "am-848586d8db-b48f9" 16:13:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG }, 16:13:12 DEBUG }, 16:13:12 DEBUG "data": { 16:13:12 DEBUG "value": [ 16:13:12 DEBUG "value": [ 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "result": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG { 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "metric": { 16:13:12 DEBUG [...] 16:13:12 DEBUG [...] 16:13:12 DEBUG "instance": "10.142.0.107:9100" 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG }, 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:13:12 DEBUG "pod": "am-848586d8db-hl5g5" 16:13:12 DEBUG 16:13:12 DEBUG }, 16:13:12 DEBUG }, 16:13:12 DEBUG 16:13:12 DEBUG "value": [ 16:13:12 DEBUG "value": [ 16:13:12 DEBUG [...] 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG 16:13:12 DEBUG "1287.0666666666666" 16:13:12 DEBUG "0" 16:13:12 DEBUG 16:13:12 DEBUG ] 16:13:12 DEBUG ] 16:13:12 DEBUG 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 DEBUG "instance": "10.142.0.43:9100" 16:13:12 DEBUG ] 16:13:12 DEBUG ] 16:13:12 DEBUG }, 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 DEBUG "value": [ 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "1616166149120" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "pod": "am-848586d8db-b48f9" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "1287.4" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539592" 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG { 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "pod": "ds-cts-0" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "pod": "ds-cts-2" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "0" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "pod": "am-848586d8db-hl5g5" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "667.5902534430771" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "pod": "ds-idrepo-0" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "pod": "ds-idrepo-2" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "0" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "instance": "10.142.0.107:9100" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "instance": "10.142.0.43:9100" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "0.03626666666661057" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713539592" 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "instance": "10.142.0.107:9100" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "instance": "10.142.0.43:9100" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "42685683.099999994" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO 16:13:12 INFO [loop_until]: kubectl --namespace=xlou top pods 16:13:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "8339570688" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539592" 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "instance": "10.142.0.107:9100" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "instance": "10.142.0.43:9100" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "42683002.5" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG ----- output ----- 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG --- status code --- 16:13:12 INFO [http_cmd]: http status code OK 16:13:12 DEBUG { 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- status code --- 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:13:12 DEBUG "data": { 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:13:12 DEBUG --- http response --- 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG "result": [ 16:13:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG { 16:13:12 DEBUG ----- output ----- 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "metric": { 16:13:12 DEBUG { 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "instance": "10.142.0.107:9100" 16:13:12 DEBUG "status": "success", 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG }, 16:13:12 DEBUG "data": { 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG "value": [ 16:13:12 DEBUG "resultType": "vector", 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG 16:13:12 DEBUG "result": [ 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG 16:13:12 DEBUG { 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "pod": "am-848586d8db-b48f9" 16:13:12 DEBUG 16:13:12 DEBUG "metric": { 16:13:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:13:12 DEBUG }, 16:13:12 DEBUG [...] 16:13:12 DEBUG "pod": "ds-cts-0" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "instance": "10.142.0.43:9100" 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG }, 16:13:12 DEBUG 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG "value": [ 16:13:12 DEBUG [...] 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "0.026000000000006667" 16:13:12 DEBUG 16:13:12 DEBUG 16:13:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:13:12 DEBUG ] 16:13:12 DEBUG 16:13:12 DEBUG "pod": "am-848586d8db-hl5g5" 16:13:12 DEBUG }, 16:13:12 DEBUG } 16:13:12 DEBUG "pod": "ds-cts-2" 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG ] 16:13:12 DEBUG }, 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG } 16:13:12 DEBUG "value": [ 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "0" 16:13:12 DEBUG } 16:13:12 DEBUG 1713539592, 16:13:12 DEBUG "69262.48879471925" 16:13:12 DEBUG ] 16:13:12 DEBUG "0.00006666666666666667" 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG ] 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 DEBUG } 16:13:12 INFO 16:13:12 INFO [loop_until]: kubectl --namespace=xlou top node 16:13:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:13:12 INFO [loop_until]: OK (rc = 0) 16:13:12 DEBUG --- stdout --- 16:13:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-848586d8db-b48f9 8008m 7191Mi am-848586d8db-f4ck2 9718m 8848Mi am-848586d8db-hl5g5 8423m 7101Mi ds-cts-0 7605m 11349Mi ds-cts-1 8568m 11336Mi ds-cts-2 8944m 10325Mi ds-idrepo-0 3529m 3387Mi ds-idrepo-1 3583m 3277Mi ds-idrepo-2 3465m 3199Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1173Mi idm-b85f9d978-wnhxb 4m 1216Mi lodemon-69554696-dct8l 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7934m 7852Mi 16:13:12 DEBUG --- stderr --- 16:13:12 DEBUG 16:13:12 INFO [loop_until]: OK (rc = 0) 16:13:12 DEBUG --- stdout --- 16:13:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6193Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3562m 11% 6943Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8208m 27% 9818Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 8126m 27% 10383Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 7352m 24% 14360Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 5205Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 55m 0% 2747Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 19066m 63% 12590Mi 11% gke-xlou-cdm-ds-0f601751-hmm7 52m 0% 2183Mi 1% gke-xlou-cdm-ds-0f601751-j488 3758m 12% 6419Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8712m 29% 16072Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3717m 12% 7093Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 8401m 28% 12774Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8244m 51% 11011Mi 18% 16:13:12 DEBUG --- stderr --- 16:13:12 DEBUG 16:13:52 WARNING File /tmp/lodemon_data-20240419_160702/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [19/Apr/2024 16:13:52] "GET /monitoring/average?start_time=20240419_151014&stop_time=20240419_151151 HTTP/1.1" 200 - 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "am-848586d8db-b48f9" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "219467" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "am-848586d8db-b48f9" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "1396.4" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 ERROR [http_cmd]: ERROR 16:14:12 ERROR --- status code --- 16:14:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 16:14:12 ERROR --- http response --- 16:14:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 16:14:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713539652" 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "ds-cts-0" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "ds-cts-2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "0" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "ds-idrepo-0" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "ds-idrepo-2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "0" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "8.022380198825774" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "ds-idrepo-0" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "ds-idrepo-2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "0" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "ds-idrepo-0" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "ds-cts-2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "0" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "ds-idrepo-2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "2.0427598774065903" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG { 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "data": { 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG "result": [ 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG { 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:14:12 DEBUG "pod": "am-848586d8db-b48f9" 16:14:12 DEBUG "pod": "am-848586d8db-b48f9" 16:14:12 DEBUG }, 16:14:12 DEBUG }, 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG "value": [ 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG [...] 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "ds-idrepo-2" 16:14:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:14:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:14:12 DEBUG }, 16:14:12 DEBUG }, 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG "value": [ 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "0" 16:14:12 DEBUG "0.002" 16:14:12 DEBUG "0" 16:14:12 DEBUG ] 16:14:12 DEBUG ] 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG ] 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG { 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "am-848586d8db-b48f9" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "0" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "instance": "10.142.0.107:9100" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "instance": "10.142.0.43:9100" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "1617356813312" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "am-848586d8db-b48f9" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "0" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539652" 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "ds-idrepo-2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "4502.898261043374" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "ds-idrepo-0" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "ds-idrepo-2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "0" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "ds-cts-0" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "ds-cts-2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "0" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "instance": "10.142.0.107:9100" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "instance": "10.142.0.43:9100" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "0.014799999999956032" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713539652" 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "instance": "10.142.0.107:9100" 16:14:12 DEBUG "instance": "10.142.0.107:9100" 16:14:12 DEBUG }, 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "instance": "10.142.0.43:9100" 16:14:12 DEBUG "instance": "10.142.0.43:9100" 16:14:12 DEBUG }, 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "10025744.566666666" 16:14:12 DEBUG "10027461.133333333" 16:14:12 DEBUG ] 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "8665939968" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO 16:14:12 INFO [loop_until]: kubectl --namespace=xlou top pods 16:14:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539652" 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "ds-cts-0" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "ds-cts-2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "0.000033333333333333335" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "am-848586d8db-b48f9" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "0" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "instance": "10.142.0.107:9100" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "instance": "10.142.0.43:9100" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "0.011333333333336288" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO [http_cmd]: http status code OK 16:14:12 DEBUG --- status code --- 16:14:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:14:12 DEBUG --- http response --- 16:14:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 16:14:12 DEBUG ----- output ----- 16:14:12 DEBUG { 16:14:12 DEBUG "status": "success", 16:14:12 DEBUG "data": { 16:14:12 DEBUG "resultType": "vector", 16:14:12 DEBUG "result": [ 16:14:12 DEBUG { 16:14:12 DEBUG "metric": { 16:14:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG [...] 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG 16:14:12 DEBUG "pod": "ds-idrepo-2" 16:14:12 DEBUG }, 16:14:12 DEBUG "value": [ 16:14:12 DEBUG 1713539652, 16:14:12 DEBUG "137439643.94696516" 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG ] 16:14:12 DEBUG } 16:14:12 DEBUG } 16:14:12 INFO 16:14:12 INFO [loop_until]: kubectl --namespace=xlou top node 16:14:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:14:12 INFO [loop_until]: OK (rc = 0) 16:14:12 DEBUG --- stdout --- 16:14:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-848586d8db-b48f9 2043m 7193Mi am-848586d8db-f4ck2 3471m 8935Mi am-848586d8db-hl5g5 3834m 7090Mi ds-cts-0 5177m 11344Mi ds-cts-1 4197m 11332Mi ds-cts-2 4513m 10326Mi ds-idrepo-0 975m 3386Mi ds-idrepo-1 907m 3278Mi ds-idrepo-2 885m 3200Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-69554696-dct8l 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 2608m 15Mi 16:14:12 DEBUG --- stderr --- 16:14:12 DEBUG 16:14:12 INFO [loop_until]: OK (rc = 0) 16:14:12 DEBUG --- stdout --- 16:14:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6211Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 803m 2% 6936Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 3343m 11% 9823Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 3034m 10% 10395Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 3810m 12% 14360Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 140m 0% 5198Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 54m 0% 2754Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 5994m 20% 12432Mi 11% gke-xlou-cdm-ds-0f601751-hmm7 52m 0% 2179Mi 1% gke-xlou-cdm-ds-0f601751-j488 1422m 4% 6416Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 4821m 16% 16056Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 814m 2% 7077Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 4315m 14% 12773Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 2433m 15% 3851Mi 6% 16:14:12 DEBUG --- stderr --- 16:14:12 DEBUG 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713539712" 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG { 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "pod": "am-848586d8db-b48f9" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:15:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG }, 16:15:12 DEBUG { 16:15:12 DEBUG "value": [ 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "data": { 16:15:12 DEBUG "1369.9" 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG ] 16:15:12 DEBUG "result": [ 16:15:12 DEBUG } 16:15:12 DEBUG { 16:15:12 DEBUG ] 16:15:12 DEBUG "metric": { 16:15:12 DEBUG } 16:15:12 DEBUG "pod": "am-848586d8db-b48f9" 16:15:12 DEBUG } 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "176425" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "pod": "ds-cts-0" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "pod": "ds-cts-2" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "0" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 ERROR [http_cmd]: ERROR 16:15:12 ERROR --- status code --- 16:15:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 16:15:12 ERROR --- http response --- 16:15:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 16:15:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713539712" 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "pod": "ds-idrepo-0" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "pod": "ds-idrepo-2" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "0" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "pod": "ds-idrepo-0" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "pod": "ds-idrepo-2" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "0" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "3.235714593299318" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "pod": "ds-idrepo-0" 16:15:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:15:12 DEBUG }, 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG { 16:15:12 DEBUG 16:15:12 DEBUG "pod": "ds-cts-2" 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "pod": "am-848586d8db-hl5g5" 16:15:12 DEBUG }, 16:15:12 DEBUG "data": { 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "result": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "0" 16:15:12 DEBUG { 16:15:12 DEBUG "9.180380903198238" 16:15:12 DEBUG ] 16:15:12 DEBUG "metric": { 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG "pod": "am-848586d8db-b48f9" 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG }, 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG "value": [ 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 DEBUG 16:15:12 DEBUG } 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "1799.5333333333333" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "pod": "am-848586d8db-b48f9" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "1799.6666666666667" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "pod": "am-848586d8db-b48f9" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "1799.2333333333333" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:15:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:15:12 DEBUG }, 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "pod": "am-848586d8db-hl5g5" 16:15:12 DEBUG "pod": "am-848586d8db-hl5g5" 16:15:12 DEBUG }, 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "0" 16:15:12 DEBUG "583.5173445402094" 16:15:12 DEBUG ] 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "pod": "am-848586d8db-b48f9" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "0.002" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539712" 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "instance": "10.142.0.107:9100" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "instance": "10.142.0.43:9100" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "1619134284800" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "pod": "ds-idrepo-0" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "pod": "ds-idrepo-2" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "0" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "instance": "10.142.0.107:9100" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "instance": "10.142.0.43:9100" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "0.11069999999994971" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "pod": "ds-cts-0" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "pod": "ds-cts-2" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "0" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539712" 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "pod": "am-848586d8db-b48f9" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539712" 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "0" 16:15:12 DEBUG ] 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713539712" 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "instance": "10.142.0.107:9100" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "instance": "10.142.0.43:9100" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "51115965.3" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "instance": "10.142.0.107:9100" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "instance": "10.142.0.43:9100" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "51116278.5" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "571662336" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539712" 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "pod": "ds-cts-0" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "pod": "ds-cts-2" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "0.00016464666666666665" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "instance": "10.142.0.107:9100" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "instance": "10.142.0.43:9100" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "0.06699999999999778" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO [http_cmd]: http status code OK 16:15:12 DEBUG --- status code --- 16:15:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:15:12 DEBUG --- http response --- 16:15:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:15:12 DEBUG ----- output ----- 16:15:12 DEBUG { 16:15:12 DEBUG "status": "success", 16:15:12 DEBUG "data": { 16:15:12 DEBUG "resultType": "vector", 16:15:12 DEBUG "result": [ 16:15:12 DEBUG { 16:15:12 DEBUG "metric": { 16:15:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG [...] 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG 16:15:12 DEBUG "pod": "am-848586d8db-hl5g5" 16:15:12 DEBUG }, 16:15:12 DEBUG "value": [ 16:15:12 DEBUG 1713539712, 16:15:12 DEBUG "60685.80383218177" 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG ] 16:15:12 DEBUG } 16:15:12 DEBUG } 16:15:12 INFO 16:15:12 INFO [loop_until]: kubectl --namespace=xlou top pods 16:15:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:15:12 INFO [loop_until]: OK (rc = 0) 16:15:12 DEBUG --- stdout --- 16:15:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-848586d8db-b48f9 9496m 7194Mi am-848586d8db-f4ck2 10178m 8947Mi am-848586d8db-hl5g5 6944m 7094Mi ds-cts-0 7199m 11353Mi ds-cts-1 9548m 11327Mi ds-cts-2 8271m 10334Mi ds-idrepo-0 4367m 3387Mi ds-idrepo-1 3992m 3279Mi ds-idrepo-2 4237m 3200Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 4m 1216Mi lodemon-69554696-dct8l 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 5953m 2178Mi 16:15:12 DEBUG --- stderr --- 16:15:12 DEBUG 16:15:12 INFO 16:15:12 INFO [loop_until]: kubectl --namespace=xlou top node 16:15:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:15:12 INFO [loop_until]: OK (rc = 0) 16:15:12 DEBUG --- stdout --- 16:15:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 6220Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4638m 15% 6952Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9283m 31% 9805Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 9608m 32% 10378Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 8331m 27% 14363Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 5204Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 53m 0% 2753Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 23053m 77% 12475Mi 11% gke-xlou-cdm-ds-0f601751-hmm7 50m 0% 2173Mi 1% gke-xlou-cdm-ds-0f601751-j488 4161m 13% 6413Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8899m 29% 16055Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 4934m 16% 7082Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9132m 30% 12785Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 6406m 40% 5888Mi 10% 16:15:12 DEBUG --- stderr --- 16:15:12 DEBUG 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "pod": "am-848586d8db-b48f9" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "1286.7" 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 ERROR [http_cmd]: ERROR 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG --- status code --- 16:16:12 ERROR --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG --- http response --- 16:16:12 ERROR --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:16:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG ----- output ----- 16:16:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 16:16:12 DEBUG { 16:16:12 DEBUG { 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "data": { 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG "result": [ 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG { 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "pod": "ds-idrepo-0" 16:16:12 DEBUG "pod": "am-848586d8db-b48f9" 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:16:12 DEBUG "pod": "ds-cts-0" 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG }, 16:16:12 DEBUG }, 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG }, 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG "value": [ 16:16:12 DEBUG "value": [ 16:16:12 DEBUG { 16:16:12 DEBUG "value": [ 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "data": { 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:16:12 DEBUG 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG [...] 16:16:12 DEBUG "result": [ 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG { 16:16:12 DEBUG 16:16:12 DEBUG { 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG 16:16:12 DEBUG "metric": { 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "data": { 16:16:12 DEBUG 16:16:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:16:12 DEBUG 16:16:12 DEBUG "pod": "ds-idrepo-2" 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:16:12 DEBUG }, 16:16:12 DEBUG "pod": "ds-cts-2" 16:16:12 DEBUG }, 16:16:12 DEBUG "result": [ 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG { 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "metric": { 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "0" 16:16:12 DEBUG "pod": "ds-idrepo-0" 16:16:12 DEBUG "184555" 16:16:12 DEBUG 16:16:12 DEBUG "0" 16:16:12 DEBUG ] 16:16:12 DEBUG }, 16:16:12 DEBUG ] 16:16:12 DEBUG [...] 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG "value": [ 16:16:12 DEBUG } 16:16:12 DEBUG 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG 16:16:12 DEBUG ] 16:16:12 DEBUG 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG 16:16:12 DEBUG } 16:16:12 DEBUG 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 DEBUG 16:16:12 DEBUG } 16:16:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:16:12 DEBUG } 16:16:12 DEBUG [...] 16:16:12 DEBUG }, 16:16:12 DEBUG 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG 16:16:12 DEBUG "8.01324215212087" 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG "pod": "ds-idrepo-2" 16:16:12 DEBUG ] 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG }, 16:16:12 DEBUG } 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG "value": [ 16:16:12 DEBUG ] 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG } 16:16:12 DEBUG "0" 16:16:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG { 16:16:12 DEBUG { 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "data": { 16:16:12 DEBUG "data": { 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG "result": [ 16:16:12 DEBUG "result": [ 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG { 16:16:12 DEBUG { 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "pod": "am-848586d8db-b48f9" 16:16:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:16:12 DEBUG "pod": "ds-idrepo-0" 16:16:12 DEBUG "pod": "am-848586d8db-b48f9" 16:16:12 DEBUG }, 16:16:12 DEBUG }, 16:16:12 DEBUG }, 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG "value": [ 16:16:12 DEBUG "value": [ 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG [...] 16:16:12 DEBUG [...] 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:16:12 DEBUG "pod": "ds-cts-2" 16:16:12 DEBUG "pod": "ds-cts-2" 16:16:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:16:12 DEBUG }, 16:16:12 DEBUG }, 16:16:12 DEBUG }, 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG "value": [ 16:16:12 DEBUG "value": [ 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "1344.6333333333332" 16:16:12 DEBUG "3.860557975071944" 16:16:12 DEBUG "0" 16:16:12 DEBUG "1344.6333333333334" 16:16:12 DEBUG ] 16:16:12 DEBUG ] 16:16:12 DEBUG ] 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG ] 16:16:12 DEBUG ] 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "pod": "am-848586d8db-b48f9" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "1344.6333333333332" 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539772" 16:16:12 DEBUG ----- output ----- 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG { 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "data": { 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:16:12 DEBUG { 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG "metric": { 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG { 16:16:12 DEBUG "pod": "am-848586d8db-b48f9" 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "instance": "10.142.0.107:9100" 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG }, 16:16:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:16:12 DEBUG }, 16:16:12 DEBUG "data": { 16:16:12 DEBUG "value": [ 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG "result": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG { 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "metric": { 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG }, 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:16:12 DEBUG 16:16:12 DEBUG "instance": "10.142.0.43:9100" 16:16:12 DEBUG 16:16:12 DEBUG }, 16:16:12 DEBUG "pod": "ds-cts-2" 16:16:12 DEBUG }, 16:16:12 DEBUG 16:16:12 DEBUG "value": [ 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG [...] 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG 16:16:12 DEBUG "0.003" 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "0.14203333333332654" 16:16:12 DEBUG 16:16:12 DEBUG ] 16:16:12 DEBUG "0" 16:16:12 DEBUG ] 16:16:12 DEBUG 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG "pod": "ds-cts-2" 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG }, 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG "value": [ 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG } 16:16:12 DEBUG "12591406.42092134" 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "instance": "10.142.0.107:9100" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "instance": "10.142.0.43:9100" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "1621435292672" 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "pod": "ds-idrepo-0" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "pod": "ds-idrepo-2" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "0" 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "pod": "ds-cts-0" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "pod": "ds-cts-2" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "0" 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "pod": "am-848586d8db-b48f9" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "0" 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539772" 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "instance": "10.142.0.107:9100" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "instance": "10.142.0.43:9100" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "0.07399999999999711" 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713539772" 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "instance": "10.142.0.107:9100" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "instance": "10.142.0.43:9100" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "42856516.5" 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "instance": "10.142.0.107:9100" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "instance": "10.142.0.43:9100" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "42857196.7" 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "3707482112" 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539772" 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "pod": "ds-cts-0" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "pod": "ds-cts-2" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "0.000033333333333333335" 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO [http_cmd]: http status code OK 16:16:12 DEBUG --- status code --- 16:16:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:16:12 DEBUG --- http response --- 16:16:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:16:12 DEBUG ----- output ----- 16:16:12 DEBUG { 16:16:12 DEBUG "status": "success", 16:16:12 DEBUG "data": { 16:16:12 DEBUG "resultType": "vector", 16:16:12 DEBUG "result": [ 16:16:12 DEBUG { 16:16:12 DEBUG "metric": { 16:16:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG [...] 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG 16:16:12 DEBUG "pod": "ds-cts-2" 16:16:12 DEBUG }, 16:16:12 DEBUG "value": [ 16:16:12 DEBUG 1713539772, 16:16:12 DEBUG "355927.58897606103" 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG ] 16:16:12 DEBUG } 16:16:12 DEBUG } 16:16:12 INFO 16:16:12 INFO [loop_until]: kubectl --namespace=xlou top pods 16:16:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:16:12 INFO [loop_until]: OK (rc = 0) 16:16:12 DEBUG --- stdout --- 16:16:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-848586d8db-b48f9 7829m 7192Mi am-848586d8db-f4ck2 8945m 8945Mi am-848586d8db-hl5g5 8348m 7100Mi ds-cts-0 7001m 11359Mi ds-cts-1 7342m 11347Mi ds-cts-2 7635m 10352Mi ds-idrepo-0 3633m 3387Mi ds-idrepo-1 3621m 3278Mi ds-idrepo-2 3533m 3200Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 3m 1173Mi idm-b85f9d978-wnhxb 4m 1216Mi lodemon-69554696-dct8l 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8076m 5240Mi 16:16:12 DEBUG --- stderr --- 16:16:12 DEBUG 16:16:12 INFO 16:16:12 INFO [loop_until]: kubectl --namespace=xlou top node 16:16:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:16:12 INFO [loop_until]: OK (rc = 0) 16:16:12 DEBUG --- stdout --- 16:16:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 6213Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3678m 12% 6959Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8285m 27% 9815Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 8162m 27% 10396Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 6625m 22% 14358Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 151m 0% 5188Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 52m 0% 2758Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 18676m 62% 12679Mi 11% gke-xlou-cdm-ds-0f601751-hmm7 51m 0% 2172Mi 1% gke-xlou-cdm-ds-0f601751-j488 3707m 12% 6412Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 7493m 25% 16069Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3713m 12% 7087Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 7734m 25% 12797Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8196m 51% 8733Mi 14% 16:16:12 DEBUG --- stderr --- 16:16:12 DEBUG 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "pod": "am-848586d8db-b48f9" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "0" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713539832" 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 ERROR [http_cmd]: ERROR 16:17:12 ERROR --- status code --- 16:17:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 16:17:12 ERROR --- http response --- 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG --- status code --- 16:17:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "pod": "am-848586d8db-b48f9" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "207328.5" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "pod": "ds-idrepo-0" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "pod": "ds-idrepo-2" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "0" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "pod": "ds-idrepo-0" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "pod": "ds-idrepo-2" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "0" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG "data": { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG "data": { 16:17:12 DEBUG { 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "metric": { 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "result": [ 16:17:12 DEBUG "pod": "ds-cts-0" 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "pod": "am-848586d8db-b48f9" 16:17:12 DEBUG { 16:17:12 DEBUG }, 16:17:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:17:12 DEBUG }, 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "value": [ 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:17:12 DEBUG 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG }, 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "pod": "ds-cts-2" 16:17:12 DEBUG 16:17:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:17:12 DEBUG 16:17:12 DEBUG }, 16:17:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 16:17:12 DEBUG }, 16:17:12 DEBUG 16:17:12 DEBUG "value": [ 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG }, 16:17:12 DEBUG "0" 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "1406.8333333333333" 16:17:12 DEBUG "value": [ 16:17:12 DEBUG ] 16:17:12 DEBUG "0.002826128233817967" 16:17:12 DEBUG ] 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG "8.026981551371554" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539832" 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG --- http response --- 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG { 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "data": { 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG "result": [ 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG { 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "instance": "10.142.0.107:9100" 16:17:12 DEBUG "pod": "ds-idrepo-0" 16:17:12 DEBUG "pod": "am-848586d8db-b48f9" 16:17:12 DEBUG }, 16:17:12 DEBUG }, 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG "value": [ 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG [...] 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "instance": "10.142.0.43:9100" 16:17:12 DEBUG "pod": "ds-cts-2" 16:17:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:17:12 DEBUG }, 16:17:12 DEBUG }, 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG "value": [ 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "0.03173333333341096" 16:17:12 DEBUG "0" 16:17:12 DEBUG "1405.7666666666667" 16:17:12 DEBUG ] 16:17:12 DEBUG ] 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG ] 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "pod": "am-848586d8db-b48f9" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "1411.1666666666667" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "pod": "ds-cts-0" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "pod": "ds-cts-2" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "0" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539832" 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "pod": "am-848586d8db-b48f9" 16:17:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:17:12 DEBUG }, 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG [...] 16:17:12 DEBUG [...] 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:17:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG }, 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG }, 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG "value": [ 16:17:12 DEBUG { 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "0.003" 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG "data": { 16:17:12 DEBUG "0" 16:17:12 DEBUG ] 16:17:12 DEBUG { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG "data": { 16:17:12 DEBUG { 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "metric": { 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG "result": [ 16:17:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:17:12 DEBUG } 16:17:12 DEBUG { 16:17:12 DEBUG }, 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "value": [ 16:17:12 DEBUG "instance": "10.142.0.107:9100" 16:17:12 DEBUG 16:17:12 DEBUG }, 16:17:12 DEBUG 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 16:17:12 DEBUG 16:17:12 DEBUG }, 16:17:12 DEBUG 16:17:12 DEBUG "value": [ 16:17:12 DEBUG "instance": "10.142.0.43:9100" 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG }, 16:17:12 DEBUG "416.64123690367205" 16:17:12 DEBUG "value": [ 16:17:12 DEBUG ] 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG } 16:17:12 DEBUG "1623067087872" 16:17:12 DEBUG ] 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539832" 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "pod": "ds-idrepo-0" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "pod": "ds-idrepo-2" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "0" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "instance": "10.142.0.107:9100" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "instance": "10.142.0.43:9100" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "34072292.166666664" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "instance": "10.142.0.107:9100" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "instance": "10.142.0.43:9100" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "0.022666666666666887" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "pod": "am-848586d8db-b48f9" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "0" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713539832" 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "instance": "10.142.0.107:9100" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "instance": "10.142.0.43:9100" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "34071384.53333333" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "pod": "ds-cts-0" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "pod": "ds-cts-2" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "0.000033333333333333335" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "7135039488" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539832" 16:17:12 INFO [http_cmd]: http status code OK 16:17:12 DEBUG --- status code --- 16:17:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:17:12 DEBUG --- http response --- 16:17:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:17:12 DEBUG ----- output ----- 16:17:12 DEBUG { 16:17:12 DEBUG "status": "success", 16:17:12 DEBUG "data": { 16:17:12 DEBUG "resultType": "vector", 16:17:12 DEBUG "result": [ 16:17:12 DEBUG { 16:17:12 DEBUG "metric": { 16:17:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG [...] 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG 16:17:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 16:17:12 DEBUG }, 16:17:12 DEBUG "value": [ 16:17:12 DEBUG 1713539832, 16:17:12 DEBUG "43440126.40287526" 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG ] 16:17:12 DEBUG } 16:17:12 DEBUG } 16:17:12 INFO 16:17:12 INFO [loop_until]: kubectl --namespace=xlou top pods 16:17:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:17:12 INFO [loop_until]: OK (rc = 0) 16:17:12 DEBUG --- stdout --- 16:17:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-848586d8db-b48f9 7225m 8808Mi am-848586d8db-f4ck2 8032m 8938Mi am-848586d8db-hl5g5 6790m 8819Mi ds-cts-0 4682m 11369Mi ds-cts-1 5432m 11353Mi ds-cts-2 4288m 10353Mi ds-idrepo-0 3312m 3387Mi ds-idrepo-1 3305m 3278Mi ds-idrepo-2 3182m 3200Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1173Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-69554696-dct8l 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8001m 6829Mi 16:17:12 DEBUG --- stderr --- 16:17:12 DEBUG 16:17:12 INFO 16:17:12 INFO [loop_until]: kubectl --namespace=xlou top node 16:17:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:17:12 INFO [loop_until]: OK (rc = 0) 16:17:12 DEBUG --- stdout --- 16:17:12 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6213Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3345m 11% 6950Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6957m 23% 11522Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 6997m 23% 11990Mi 10% gke-xlou-cdm-default-pool-1b0769c8-w15q 4633m 15% 14379Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 149m 0% 5210Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 55m 0% 2741Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 18070m 60% 12666Mi 11% gke-xlou-cdm-ds-0f601751-hmm7 51m 0% 2174Mi 1% gke-xlou-cdm-ds-0f601751-j488 3370m 11% 6420Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 4737m 15% 16085Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3522m 11% 7082Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 3957m 13% 12805Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8234m 51% 10192Mi 17% 16:17:12 DEBUG --- stderr --- 16:17:12 DEBUG 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "pod": "am-848586d8db-b48f9" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "1803.8666666666666" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713539892" 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "pod": "ds-cts-0" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "pod": "ds-cts-2" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "0" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 ERROR [http_cmd]: ERROR 16:18:12 ERROR --- status code --- 16:18:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 16:18:12 ERROR --- http response --- 16:18:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 16:18:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713539892" 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "pod": "ds-idrepo-0" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "pod": "ds-idrepo-2" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "0" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:18:12 DEBUG { 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG { 16:18:12 DEBUG "data": { 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG "data": { 16:18:12 DEBUG "result": [ 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG { 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG "result": [ 16:18:12 DEBUG "metric": { 16:18:12 DEBUG { 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:18:12 DEBUG "pod": "ds-idrepo-0" 16:18:12 DEBUG "metric": { 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG }, 16:18:12 DEBUG "pod": "am-848586d8db-b48f9" 16:18:12 DEBUG { 16:18:12 DEBUG "value": [ 16:18:12 DEBUG }, 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG 16:18:12 DEBUG "value": [ 16:18:12 DEBUG "data": { 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "result": [ 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG { 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG "metric": { 16:18:12 DEBUG 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG 16:18:12 DEBUG "pod": "am-848586d8db-b48f9" 16:18:12 DEBUG 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG 16:18:12 DEBUG }, 16:18:12 DEBUG "pod": "ds-idrepo-2" 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG 16:18:12 DEBUG "value": [ 16:18:12 DEBUG }, 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:18:12 DEBUG 16:18:12 DEBUG "value": [ 16:18:12 DEBUG }, 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:18:12 DEBUG 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG 16:18:12 DEBUG "0" 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG { 16:18:12 DEBUG [...] 16:18:12 DEBUG ] 16:18:12 DEBUG "1305.5666666666666" 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG "data": { 16:18:12 DEBUG 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG "result": [ 16:18:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 DEBUG { 16:18:12 DEBUG }, 16:18:12 DEBUG } 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "value": [ 16:18:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG }, 16:18:12 DEBUG "247958.5" 16:18:12 DEBUG "value": [ 16:18:12 DEBUG ] 16:18:12 DEBUG 16:18:12 DEBUG } 16:18:12 DEBUG 16:18:12 DEBUG ] 16:18:12 DEBUG 16:18:12 DEBUG } 16:18:12 DEBUG [...] 16:18:12 DEBUG } 16:18:12 DEBUG 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG "8.034921741600854" 16:18:12 DEBUG { 16:18:12 DEBUG ] 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG } 16:18:12 DEBUG "data": { 16:18:12 DEBUG ] 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG } 16:18:12 DEBUG "result": [ 16:18:12 DEBUG } 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "0.003007829595187145" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539892" 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "instance": "10.142.0.107:9100" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "instance": "10.142.0.43:9100" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "0.14726666666666877" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "pod": "am-848586d8db-b48f9" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:18:12 DEBUG 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:18:12 DEBUG { 16:18:12 DEBUG }, 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "value": [ 16:18:12 DEBUG "data": { 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "1305.9" 16:18:12 DEBUG "result": [ 16:18:12 DEBUG ] 16:18:12 DEBUG { 16:18:12 DEBUG } 16:18:12 DEBUG "metric": { 16:18:12 DEBUG ] 16:18:12 DEBUG "pod": "ds-idrepo-0" 16:18:12 DEBUG } 16:18:12 DEBUG }, 16:18:12 DEBUG } 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "pod": "ds-cts-2" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "0" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "pod": "am-848586d8db-b48f9" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "1305.4666666666667" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "pod": "ds-cts-0" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "pod": "ds-cts-2" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "0" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "420.1312910284464" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539892" 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "instance": "10.142.0.107:9100" 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "pod": "am-848586d8db-b48f9" 16:18:12 DEBUG }, 16:18:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "instance": "10.142.0.43:9100" 16:18:12 DEBUG 16:18:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:18:12 DEBUG }, 16:18:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "1624742007808" 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "0.004" 16:18:12 DEBUG ] 16:18:12 DEBUG "0" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539892" 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539892" 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "pod": "ds-idrepo-0" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "pod": "ds-idrepo-2" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "0" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "instance": "10.142.0.107:9100" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "instance": "10.142.0.43:9100" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "43444203.5" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "instance": "10.142.0.107:9100" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "instance": "10.142.0.43:9100" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "0.07466666666665939" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "pod": "ds-cts-0" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "pod": "ds-cts-2" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "0.000033333333333333335" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "pod": "am-848586d8db-b48f9" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "0" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539892" 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "instance": "10.142.0.107:9100" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "instance": "10.142.0.43:9100" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "43447126.266666666" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713539892" 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "8238125056" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539892" 16:18:12 INFO [http_cmd]: http status code OK 16:18:12 DEBUG --- status code --- 16:18:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:18:12 DEBUG --- http response --- 16:18:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:18:12 DEBUG ----- output ----- 16:18:12 DEBUG { 16:18:12 DEBUG "status": "success", 16:18:12 DEBUG "data": { 16:18:12 DEBUG "resultType": "vector", 16:18:12 DEBUG "result": [ 16:18:12 DEBUG { 16:18:12 DEBUG "metric": { 16:18:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG [...] 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG 16:18:12 DEBUG "pod": "idm-b85f9d978-wnhxb" 16:18:12 DEBUG }, 16:18:12 DEBUG "value": [ 16:18:12 DEBUG 1713539892, 16:18:12 DEBUG "15264.77024070022" 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG ] 16:18:12 DEBUG } 16:18:12 DEBUG } 16:18:12 INFO 16:18:12 INFO [loop_until]: kubectl --namespace=xlou top pods 16:18:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:18:12 INFO [loop_until]: OK (rc = 0) 16:18:12 DEBUG --- stdout --- 16:18:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-848586d8db-b48f9 8079m 8818Mi am-848586d8db-f4ck2 8808m 8950Mi am-848586d8db-hl5g5 8126m 8128Mi ds-cts-0 8182m 11359Mi ds-cts-1 8025m 11333Mi ds-cts-2 9109m 10356Mi ds-idrepo-0 3690m 3387Mi ds-idrepo-1 3477m 3278Mi ds-idrepo-2 3596m 3199Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1172Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-69554696-dct8l 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7865m 8002Mi 16:18:12 DEBUG --- stderr --- 16:18:12 DEBUG 16:18:12 INFO 16:18:12 INFO [loop_until]: kubectl --namespace=xlou top node 16:18:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:18:13 INFO [loop_until]: OK (rc = 0) 16:18:13 DEBUG --- stdout --- 16:18:13 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 6208Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3615m 12% 6943Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8144m 27% 10857Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 7862m 26% 12006Mi 10% gke-xlou-cdm-default-pool-1b0769c8-w15q 7207m 24% 14375Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5205Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 51m 0% 2745Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 18393m 61% 12695Mi 11% gke-xlou-cdm-ds-0f601751-hmm7 50m 0% 2165Mi 1% gke-xlou-cdm-ds-0f601751-j488 3608m 12% 6415Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 8319m 27% 16069Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 3849m 12% 7086Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 9287m 31% 12793Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 8301m 52% 10872Mi 18% 16:18:13 DEBUG --- stderr --- 16:18:13 DEBUG 16:18:46 WARNING File /tmp/lodemon_data-20240419_160702/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [19/Apr/2024 16:18:46] "GET /monitoring/average?start_time=20240419_151552&stop_time=20240419_151645 HTTP/1.1" 200 - 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "data": { 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "result": [ 16:19:12 DEBUG { 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "pod": "am-848586d8db-b48f9" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG [...] 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "1340.4333333333334" 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713539952" 16:19:12 ERROR [http_cmd]: ERROR 16:19:12 ERROR --- status code --- 16:19:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 16:19:12 ERROR --- http response --- 16:19:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 16:19:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713539952" 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG --- status code --- 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG --- http response --- 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG { 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG { 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG "data": { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG { 16:19:12 DEBUG { 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:19:12 DEBUG "data": { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "result": [ 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "data": { 16:19:12 DEBUG "data": { 16:19:12 DEBUG { 16:19:12 DEBUG { 16:19:12 DEBUG "result": [ 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG { 16:19:12 DEBUG "result": [ 16:19:12 DEBUG { 16:19:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:19:12 DEBUG "data": { 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "result": [ 16:19:12 DEBUG "metric": { 16:19:12 DEBUG }, 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "pod": "ds-cts-0" 16:19:12 DEBUG { 16:19:12 DEBUG "pod": "ds-idrepo-0" 16:19:12 DEBUG "value": [ 16:19:12 DEBUG "result": [ 16:19:12 DEBUG }, 16:19:12 DEBUG "metric": { 16:19:12 DEBUG }, 16:19:12 DEBUG 16:19:12 DEBUG { 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG "value": [ 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG "pod": "am-848586d8db-b48f9" 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG "metric": { 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG }, 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "pod": "am-848586d8db-b48f9" 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG "value": [ 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG 16:19:12 DEBUG [...] 16:19:12 DEBUG }, 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "value": [ 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:19:12 DEBUG [...] 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:19:12 DEBUG 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:19:12 DEBUG [...] 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG { 16:19:12 DEBUG 16:19:12 DEBUG { 16:19:12 DEBUG [...] 16:19:12 DEBUG { 16:19:12 DEBUG 16:19:12 DEBUG { 16:19:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:19:12 DEBUG 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG }, 16:19:12 DEBUG [...] 16:19:12 DEBUG "data": { 16:19:12 DEBUG "pod": "ds-cts-2" 16:19:12 DEBUG "data": { 16:19:12 DEBUG 16:19:12 DEBUG "data": { 16:19:12 DEBUG "pod": "ds-idrepo-2" 16:19:12 DEBUG "data": { 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG }, 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG }, 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG 16:19:12 DEBUG "result": [ 16:19:12 DEBUG "value": [ 16:19:12 DEBUG "result": [ 16:19:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:19:12 DEBUG "result": [ 16:19:12 DEBUG "value": [ 16:19:12 DEBUG "result": [ 16:19:12 DEBUG "8.002902024250348" 16:19:12 DEBUG 16:19:12 DEBUG { 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG { 16:19:12 DEBUG }, 16:19:12 DEBUG { 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG { 16:19:12 DEBUG ] 16:19:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "0" 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "value": [ 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "0" 16:19:12 DEBUG "metric": { 16:19:12 DEBUG } 16:19:12 DEBUG }, 16:19:12 DEBUG "pod": "am-848586d8db-b48f9" 16:19:12 DEBUG ] 16:19:12 DEBUG "pod": "ds-idrepo-0" 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "pod": "am-848586d8db-b48f9" 16:19:12 DEBUG ] 16:19:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:19:12 DEBUG ] 16:19:12 DEBUG "value": [ 16:19:12 DEBUG }, 16:19:12 DEBUG } 16:19:12 DEBUG }, 16:19:12 DEBUG "0" 16:19:12 DEBUG }, 16:19:12 DEBUG } 16:19:12 DEBUG }, 16:19:12 DEBUG } 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG ] 16:19:12 DEBUG "value": [ 16:19:12 DEBUG ] 16:19:12 DEBUG "value": [ 16:19:12 DEBUG ] 16:19:12 DEBUG "value": [ 16:19:12 DEBUG } 16:19:12 DEBUG "0" 16:19:12 DEBUG 16:19:12 DEBUG } 16:19:12 DEBUG 16:19:12 DEBUG } 16:19:12 DEBUG 16:19:12 DEBUG } 16:19:12 DEBUG 16:19:12 DEBUG ] 16:19:12 DEBUG 16:19:12 DEBUG } 16:19:12 DEBUG 16:19:12 DEBUG ] 16:19:12 DEBUG 16:19:12 DEBUG } 16:19:12 DEBUG 16:19:12 DEBUG } 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG } 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG ] 16:19:12 DEBUG [...] 16:19:12 DEBUG [...] 16:19:12 DEBUG } 16:19:12 DEBUG [...] 16:19:12 DEBUG [...] 16:19:12 DEBUG } 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG } 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:19:12 DEBUG "pod": "ds-idrepo-2" 16:19:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:19:12 DEBUG "pod": "am-848586d8db-hl5g5" 16:19:12 DEBUG }, 16:19:12 DEBUG }, 16:19:12 DEBUG }, 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG "value": [ 16:19:12 DEBUG "value": [ 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "209935.5" 16:19:12 DEBUG "0" 16:19:12 DEBUG "0" 16:19:12 DEBUG "2.961683407399472" 16:19:12 DEBUG ] 16:19:12 DEBUG ] 16:19:12 DEBUG ] 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG ] 16:19:12 DEBUG ] 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539952" 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "data": { 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "result": [ 16:19:12 DEBUG { 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG [...] 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "pod": "am-848586d8db-hl5g5" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "438.61433849119237" 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "data": { 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "result": [ 16:19:12 DEBUG { 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "instance": "10.142.0.107:9100" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG [...] 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "instance": "10.142.0.43:9100" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "0.015066666666657132" 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "data": { 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "result": [ 16:19:12 DEBUG { 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "pod": "ds-idrepo-0" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG [...] 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "pod": "ds-cts-2" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "0" 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "data": { 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "result": [ 16:19:12 DEBUG { 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "pod": "ds-cts-0" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG [...] 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "pod": "ds-cts-2" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "0" 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539952" 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539952" 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539952" 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG { 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "data": { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "data": { 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "data": { 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "result": [ 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "result": [ 16:19:12 DEBUG { 16:19:12 DEBUG "result": [ 16:19:12 DEBUG { 16:19:12 DEBUG "metric": { 16:19:12 DEBUG { 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "instance": "10.142.0.107:9100" 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:19:12 DEBUG }, 16:19:12 DEBUG "pod": "am-848586d8db-b48f9" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG [...] 16:19:12 DEBUG 16:19:12 DEBUG [...] 16:19:12 DEBUG 16:19:12 DEBUG [...] 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "instance": "10.142.0.43:9100" 16:19:12 DEBUG 16:19:12 DEBUG "pod": "am-848586d8db-hl5g5" 16:19:12 DEBUG }, 16:19:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "1626048029696" 16:19:12 DEBUG 1713539952, 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG "0" 16:19:12 DEBUG ] 16:19:12 DEBUG "0.005" 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG } 16:19:12 DEBUG { 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG "data": { 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG "result": [ 16:19:12 DEBUG { 16:19:12 DEBUG { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "data": { 16:19:12 DEBUG "instance": "10.142.0.107:9100" 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG }, 16:19:12 DEBUG "result": [ 16:19:12 DEBUG "value": [ 16:19:12 DEBUG { 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG 16:19:12 DEBUG "metric": { 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG 16:19:12 DEBUG "instance": "10.142.0.107:9100" 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG 16:19:12 DEBUG }, 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG [...] 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG { 16:19:12 DEBUG 16:19:12 DEBUG "instance": "10.142.0.43:9100" 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG [...] 16:19:12 DEBUG }, 16:19:12 DEBUG "data": { 16:19:12 DEBUG 16:19:12 DEBUG "value": [ 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "result": [ 16:19:12 DEBUG 16:19:12 DEBUG "10035771.966666667" 16:19:12 DEBUG { 16:19:12 DEBUG "instance": "10.142.0.43:9100" 16:19:12 DEBUG ] 16:19:12 DEBUG "metric": { 16:19:12 DEBUG }, 16:19:12 DEBUG } 16:19:12 DEBUG "pod": "ds-idrepo-0" 16:19:12 DEBUG "value": [ 16:19:12 DEBUG ] 16:19:12 DEBUG }, 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG } 16:19:12 DEBUG "value": [ 16:19:12 DEBUG "0.009333333333328635" 16:19:12 DEBUG } 16:19:12 DEBUG 16:19:12 DEBUG ] 16:19:12 DEBUG 16:19:12 DEBUG } 16:19:12 DEBUG 16:19:12 DEBUG ] 16:19:12 DEBUG [...] 16:19:12 DEBUG } 16:19:12 DEBUG 16:19:12 DEBUG } 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "pod": "ds-idrepo-2" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "0" 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "data": { 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "result": [ 16:19:12 DEBUG { 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "pod": "ds-cts-0" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG [...] 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "pod": "ds-cts-2" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "0.00006666666666666667" 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "data": { 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "result": [ 16:19:12 DEBUG { 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "pod": "am-848586d8db-b48f9" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG [...] 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "0" 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713539952" 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "data": { 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "result": [ 16:19:12 DEBUG { 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "instance": "10.142.0.107:9100" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG [...] 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "instance": "10.142.0.43:9100" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "9820772.5" 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713539952" 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "data": { 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "result": [ 16:19:12 DEBUG { 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG [...] 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "8580104192" 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713539952" 16:19:12 INFO [http_cmd]: http status code OK 16:19:12 DEBUG --- status code --- 16:19:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:19:12 DEBUG --- http response --- 16:19:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 16:19:12 DEBUG ----- output ----- 16:19:12 DEBUG { 16:19:12 DEBUG "status": "success", 16:19:12 DEBUG "data": { 16:19:12 DEBUG "resultType": "vector", 16:19:12 DEBUG "result": [ 16:19:12 DEBUG { 16:19:12 DEBUG "metric": { 16:19:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG [...] 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG 16:19:12 DEBUG "pod": "am-848586d8db-hl5g5" 16:19:12 DEBUG }, 16:19:12 DEBUG "value": [ 16:19:12 DEBUG 1713539952, 16:19:12 DEBUG "3618.568292552337" 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG ] 16:19:12 DEBUG } 16:19:12 DEBUG } 16:19:12 INFO 16:19:12 INFO [loop_until]: kubectl --namespace=xlou top pods 16:19:12 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:19:12 INFO [loop_until]: OK (rc = 0) 16:19:12 DEBUG --- stdout --- 16:19:12 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-848586d8db-b48f9 1352m 8327Mi am-848586d8db-f4ck2 3062m 8483Mi am-848586d8db-hl5g5 3939m 8129Mi ds-cts-0 3984m 11351Mi ds-cts-1 5449m 11333Mi ds-cts-2 4702m 10332Mi ds-idrepo-0 719m 3387Mi ds-idrepo-1 1194m 3279Mi ds-idrepo-2 1373m 3200Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 1172Mi idm-b85f9d978-wnhxb 3m 1216Mi lodemon-69554696-dct8l 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 128m 15Mi 16:19:12 DEBUG --- stderr --- 16:19:12 DEBUG 16:19:13 INFO 16:19:13 INFO [loop_until]: kubectl --namespace=xlou top node 16:19:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:19:13 INFO [loop_until]: OK (rc = 0) 16:19:13 DEBUG --- stdout --- 16:19:13 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 6218Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 468m 1% 6939Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2149m 7% 10869Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 1793m 6% 11543Mi 10% gke-xlou-cdm-default-pool-1b0769c8-w15q 4291m 14% 14366Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 151m 0% 5197Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 55m 0% 2746Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 2835m 9% 12018Mi 10% gke-xlou-cdm-ds-0f601751-hmm7 52m 0% 2163Mi 1% gke-xlou-cdm-ds-0f601751-j488 870m 2% 6416Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 4311m 14% 16077Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 488m 1% 7094Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 3917m 13% 12782Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 522m 3% 3852Mi 6% 16:19:13 DEBUG --- stderr --- 16:19:13 DEBUG 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "data": { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "pod": "am-848586d8db-b48f9" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "1411.3" 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713540012" 16:20:12 ERROR [http_cmd]: ERROR 16:20:12 ERROR --- status code --- 16:20:12 ERROR http status code is 400 (expected [200, 401, 403, 404]) 16:20:12 ERROR --- http response --- 16:20:12 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 16:20:12 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG --- status code --- 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- status code --- 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG --- status code --- 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713540012" 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- http response --- 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG { 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:20:12 DEBUG ----- output ----- 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG { 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG { 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG "data": { 16:20:12 DEBUG { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG { 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "data": { 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "data": { 16:20:12 DEBUG { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG "result": [ 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG "data": { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:20:12 DEBUG "data": { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "data": { 16:20:12 DEBUG { 16:20:12 DEBUG { 16:20:12 DEBUG { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "result": [ 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "result": [ 16:20:12 DEBUG "data": { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG { 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "result": [ 16:20:12 DEBUG "data": { 16:20:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:20:12 DEBUG "data": { 16:20:12 DEBUG { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG }, 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:20:12 DEBUG "data": { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "pod": "am-848586d8db-b48f9" 16:20:12 DEBUG { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "result": [ 16:20:12 DEBUG "value": [ 16:20:12 DEBUG "result": [ 16:20:12 DEBUG "pod": "ds-idrepo-0" 16:20:12 DEBUG }, 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "pod": "am-848586d8db-b48f9" 16:20:12 DEBUG }, 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "pod": "ds-cts-0" 16:20:12 DEBUG { 16:20:12 DEBUG 16:20:12 DEBUG { 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG "result": [ 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG "pod": "am-848586d8db-b48f9" 16:20:12 DEBUG }, 16:20:12 DEBUG "metric": { 16:20:12 DEBUG 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG { 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:20:12 DEBUG 16:20:12 DEBUG "pod": "am-848586d8db-b48f9" 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "metric": { 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG }, 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "pod": "ds-idrepo-0" 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG "value": [ 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG }, 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG "value": [ 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "pod": "lodemon-69554696-dct8l" 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:20:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG }, 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "pod": "ds-idrepo-2" 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG }, 16:20:12 DEBUG [...] 16:20:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:20:12 DEBUG }, 16:20:12 DEBUG 16:20:12 DEBUG "pod": "ds-cts-2" 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG }, 16:20:12 DEBUG { 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:20:12 DEBUG }, 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "pod": "lodemon-69554696-dct8l" 16:20:12 DEBUG 16:20:12 DEBUG "value": [ 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG "7248.347245409014" 16:20:12 DEBUG }, 16:20:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "data": { 16:20:12 DEBUG "0.027122054243316093" 16:20:12 DEBUG 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "0" 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG ] 16:20:12 DEBUG "value": [ 16:20:12 DEBUG }, 16:20:12 DEBUG "0.13333333333333333" 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG ] 16:20:12 DEBUG "pod": "ds-idrepo-2" 16:20:12 DEBUG "0" 16:20:12 DEBUG ] 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "0" 16:20:12 DEBUG } 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG ] 16:20:12 DEBUG "result": [ 16:20:12 DEBUG } 16:20:12 DEBUG }, 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG "0" 16:20:12 DEBUG ] 16:20:12 DEBUG ] 16:20:12 DEBUG "0.0011106510851418721" 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG } 16:20:12 DEBUG { 16:20:12 DEBUG ] 16:20:12 DEBUG "value": [ 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG "129802.5" 16:20:12 DEBUG ] 16:20:12 DEBUG "metric": { 16:20:12 DEBUG } 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 DEBUG } 16:20:12 DEBUG "instance": "10.142.0.107:9100" 16:20:12 DEBUG }, 16:20:12 DEBUG "0.13333333333333333" 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG "value": [ 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG 16:20:12 DEBUG } 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "instance": "10.142.0.43:9100" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "0.019699999999970904" 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "data": { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "pod": "ds-idrepo-0" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "pod": "ds-cts-2" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "0" 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713540012" 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "data": { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "pod": "ds-cts-0" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "pod": "ds-cts-2" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "0" 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- http response --- 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG { 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "data": { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "pod": "ds-cts-0" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "pod": "ds-cts-2" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "0.00006666666666666667" 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "data": { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "pod": "am-848586d8db-b48f9" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "0" 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "data": { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "instance": "10.142.0.107:9100" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "instance": "10.142.0.43:9100" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "0.013333333333332576" 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "data": { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "instance": "10.142.0.107:9100" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "instance": "10.142.0.43:9100" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "10003913.766666668" 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- http response --- 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG { 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG "data": { 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG { 16:20:12 DEBUG { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "data": { 16:20:12 DEBUG "data": { 16:20:12 DEBUG "pod": "am-848586d8db-b48f9" 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG }, 16:20:12 DEBUG "result": [ 16:20:12 DEBUG "result": [ 16:20:12 DEBUG "value": [ 16:20:12 DEBUG { 16:20:12 DEBUG { 16:20:12 DEBUG 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG 16:20:12 DEBUG "instance": "10.142.0.107:9100" 16:20:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:20:12 DEBUG 16:20:12 DEBUG }, 16:20:12 DEBUG }, 16:20:12 DEBUG [...] 16:20:12 DEBUG "value": [ 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "pod": "am-848586d8db-f4ck2" 16:20:12 DEBUG [...] 16:20:12 DEBUG [...] 16:20:12 DEBUG }, 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG 16:20:12 DEBUG "0.005" 16:20:12 DEBUG "instance": "10.142.0.43:9100" 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG "pod": "lodemon-69554696-dct8l" 16:20:12 DEBUG ] 16:20:12 DEBUG }, 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG }, 16:20:12 DEBUG } 16:20:12 DEBUG "value": [ 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG "value": [ 16:20:12 DEBUG ] 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 16:20:12 DEBUG } 16:20:12 DEBUG "1626993220608" 16:20:12 DEBUG "0" 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG ] 16:20:12 DEBUG { 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG ] 16:20:12 DEBUG ] 16:20:12 DEBUG "data": { 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "pod": "ds-idrepo-0" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "pod": "ds-idrepo-2" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "0.13333333333333333" 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713540012" 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "data": { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "instance": "10.142.0.107:9100" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "instance": "10.142.0.43:9100" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "10001699.7" 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713540012" 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "data": { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "113393664" 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:12 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713540012" 16:20:12 INFO [http_cmd]: http status code OK 16:20:12 DEBUG --- status code --- 16:20:12 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 16:20:12 DEBUG --- http response --- 16:20:12 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 16:20:12 DEBUG ----- output ----- 16:20:12 DEBUG { 16:20:12 DEBUG "status": "success", 16:20:12 DEBUG "data": { 16:20:12 DEBUG "resultType": "vector", 16:20:12 DEBUG "result": [ 16:20:12 DEBUG { 16:20:12 DEBUG "metric": { 16:20:12 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG [...] 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG 16:20:12 DEBUG "pod": "lodemon-69554696-dct8l" 16:20:12 DEBUG }, 16:20:12 DEBUG "value": [ 16:20:12 DEBUG 1713540012, 16:20:12 DEBUG "2233858.5642737895" 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG ] 16:20:12 DEBUG } 16:20:12 DEBUG } 16:20:13 INFO 16:20:13 INFO [loop_until]: kubectl --namespace=xlou top pods 16:20:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:20:13 INFO [loop_until]: OK (rc = 0) 16:20:13 DEBUG --- stdout --- 16:20:13 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-848586d8db-b48f9 1396m 8306Mi am-848586d8db-f4ck2 1520m 8379Mi am-848586d8db-hl5g5 1339m 8077Mi ds-cts-0 3677m 11353Mi ds-cts-1 4006m 11304Mi ds-cts-2 3915m 10356Mi ds-idrepo-0 1041m 3420Mi ds-idrepo-1 751m 3291Mi ds-idrepo-2 958m 3258Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 7m 1173Mi idm-b85f9d978-wnhxb 4m 1217Mi lodemon-69554696-dct8l 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1244m 179Mi 16:20:13 DEBUG --- stderr --- 16:20:13 DEBUG 16:20:13 INFO 16:20:13 INFO [loop_until]: kubectl --namespace=xlou top node 16:20:13 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 16:20:13 INFO [loop_until]: OK (rc = 0) 16:20:13 DEBUG --- stdout --- 16:20:13 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 6211Mi 5% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1094m 3% 6939Mi 6% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1369m 4% 10826Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 1532m 5% 11520Mi 10% gke-xlou-cdm-default-pool-1b0769c8-w15q 3619m 12% 14340Mi 12% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5206Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 53m 0% 2747Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1605m 5% 11829Mi 10% gke-xlou-cdm-ds-0f601751-hmm7 49m 0% 2167Mi 1% gke-xlou-cdm-ds-0f601751-j488 1019m 3% 6425Mi 5% gke-xlou-cdm-ds-0f601751-mc5d 4083m 13% 16057Mi 14% gke-xlou-cdm-ds-0f601751-nj6q 941m 3% 7063Mi 6% gke-xlou-cdm-ds-0f601751-nmxm 3953m 13% 12803Mi 11% gke-xlou-cdm-frontend-a8771548-i82n 1000m 6% 4009Mi 6% 16:20:13 DEBUG --- stderr --- 16:20:13 DEBUG 16:20:33 INFO Finished: True 16:20:33 INFO Waiting for threads to register finish flag 16:21:13 INFO Done. Have a nice day! :) 127.0.0.1 - - [19/Apr/2024 16:21:13] "GET /monitoring/stop HTTP/1.1" 200 - 16:21:16 WARNING File /tmp/lodemon_data-20240419_160702/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6481', ...) 127.0.0.1 - - [19/Apr/2024 16:21:20] "GET /monitoring/process HTTP/1.1" 200 -