==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-9886f5b6-dt5fm Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-default-pool-1b0769c8-7vmj/10.142.0.111 Start Time: Wed, 17 Apr 2024 18:15:29 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=9886f5b6 skaffold.dev/run-id=afce436d-d3f7-4da2-b79f-14d836d9ae4e Annotations: Status: Running IP: 10.106.42.182 IPs: IP: 10.106.42.182 Controlled By: ReplicaSet/lodemon-9886f5b6 Containers: lodemon: Container ID: containerd://28ebca5cfd843e18eadf55d6c624a61d3ae27b61dbc2903ba9ce628360988463 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:2febaeab396557db3e0fc07c0501fa786a35d3473828be1a84a3d1610beec5de Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Wed, 17 Apr 2024 18:15:46 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-gsksr (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-gsksr: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 14m default-scheduler Successfully assigned xlou/lodemon-9886f5b6-dt5fm to gke-xlou-cdm-default-pool-1b0769c8-7vmj Normal Pulling 14m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 14m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 16.714245313s (16.714252452s including waiting) Normal Created 14m kubelet Created container lodemon Normal Started 14m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.42.182:8080 Press CTRL+C to quit 19:16:46 INFO 19:16:46 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:46 INFO [loop_until]: OK (rc = 0) 19:16:46 DEBUG --- stdout --- 19:16:46 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:46 DEBUG --- stderr --- 19:16:46 DEBUG 19:16:46 INFO 19:16:46 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:46 INFO [loop_until]: OK (rc = 0) 19:16:46 DEBUG --- stdout --- 19:16:46 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:46 DEBUG --- stderr --- 19:16:46 DEBUG 19:16:46 INFO 19:16:46 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:46 INFO [loop_until]: OK (rc = 0) 19:16:46 DEBUG --- stdout --- 19:16:46 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:46 DEBUG --- stderr --- 19:16:46 DEBUG 19:16:46 INFO 19:16:46 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:46 INFO [loop_until]: OK (rc = 0) 19:16:46 DEBUG --- stdout --- 19:16:46 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:46 DEBUG --- stderr --- 19:16:46 DEBUG 19:16:46 INFO 19:16:46 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:46 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:46 INFO [loop_until]: OK (rc = 0) 19:16:46 DEBUG --- stdout --- 19:16:46 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:46 DEBUG --- stderr --- 19:16:46 DEBUG 19:16:47 INFO 19:16:47 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:47 INFO [loop_until]: OK (rc = 0) 19:16:47 DEBUG --- stdout --- 19:16:47 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:47 DEBUG --- stderr --- 19:16:47 DEBUG 19:16:47 INFO 19:16:47 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:47 INFO [loop_until]: OK (rc = 0) 19:16:47 DEBUG --- stdout --- 19:16:47 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:47 DEBUG --- stderr --- 19:16:47 DEBUG 19:16:47 INFO 19:16:47 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:47 INFO [loop_until]: OK (rc = 0) 19:16:47 DEBUG --- stdout --- 19:16:47 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:47 DEBUG --- stderr --- 19:16:47 DEBUG 19:16:47 INFO 19:16:47 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:47 INFO [loop_until]: OK (rc = 0) 19:16:47 DEBUG --- stdout --- 19:16:47 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:47 DEBUG --- stderr --- 19:16:47 DEBUG 19:16:47 INFO 19:16:47 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:47 INFO [loop_until]: OK (rc = 0) 19:16:47 DEBUG --- stdout --- 19:16:47 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:47 DEBUG --- stderr --- 19:16:47 DEBUG 19:16:47 INFO 19:16:47 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:47 INFO [loop_until]: OK (rc = 0) 19:16:47 DEBUG --- stdout --- 19:16:47 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:47 DEBUG --- stderr --- 19:16:47 DEBUG 19:16:47 INFO 19:16:47 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:47 INFO [loop_until]: OK (rc = 0) 19:16:47 DEBUG --- stdout --- 19:16:47 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:47 DEBUG --- stderr --- 19:16:47 DEBUG 19:16:47 INFO 19:16:47 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:47 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:48 INFO [loop_until]: OK (rc = 0) 19:16:48 DEBUG --- stdout --- 19:16:48 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:48 DEBUG --- stderr --- 19:16:48 DEBUG 19:16:48 INFO 19:16:48 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:48 INFO [loop_until]: OK (rc = 0) 19:16:48 DEBUG --- stdout --- 19:16:48 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:48 DEBUG --- stderr --- 19:16:48 DEBUG 19:16:48 INFO 19:16:48 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:48 INFO [loop_until]: OK (rc = 0) 19:16:48 DEBUG --- stdout --- 19:16:48 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:48 DEBUG --- stderr --- 19:16:48 DEBUG 19:16:48 INFO 19:16:48 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:48 INFO [loop_until]: OK (rc = 0) 19:16:48 DEBUG --- stdout --- 19:16:48 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:48 DEBUG --- stderr --- 19:16:48 DEBUG 19:16:48 INFO 19:16:48 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:48 INFO [loop_until]: OK (rc = 0) 19:16:48 DEBUG --- stdout --- 19:16:48 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:48 DEBUG --- stderr --- 19:16:48 DEBUG 19:16:48 INFO 19:16:48 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:48 INFO [loop_until]: OK (rc = 0) 19:16:48 DEBUG --- stdout --- 19:16:48 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:48 DEBUG --- stderr --- 19:16:48 DEBUG 19:16:48 INFO 19:16:48 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:48 INFO [loop_until]: OK (rc = 0) 19:16:48 DEBUG --- stdout --- 19:16:48 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:48 DEBUG --- stderr --- 19:16:48 DEBUG 19:16:48 INFO 19:16:48 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:48 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:49 INFO [loop_until]: OK (rc = 0) 19:16:49 DEBUG --- stdout --- 19:16:49 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:49 DEBUG --- stderr --- 19:16:49 DEBUG 19:16:49 INFO 19:16:49 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:49 INFO [loop_until]: OK (rc = 0) 19:16:49 DEBUG --- stdout --- 19:16:49 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:49 DEBUG --- stderr --- 19:16:49 DEBUG 19:16:49 INFO 19:16:49 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:49 INFO [loop_until]: OK (rc = 0) 19:16:49 DEBUG --- stdout --- 19:16:49 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:49 DEBUG --- stderr --- 19:16:49 DEBUG 19:16:49 INFO 19:16:49 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:49 INFO [loop_until]: OK (rc = 0) 19:16:49 DEBUG --- stdout --- 19:16:49 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:49 DEBUG --- stderr --- 19:16:49 DEBUG 19:16:49 INFO 19:16:49 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:49 INFO [loop_until]: OK (rc = 0) 19:16:49 DEBUG --- stdout --- 19:16:49 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:49 DEBUG --- stderr --- 19:16:49 DEBUG 19:16:49 INFO 19:16:49 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:49 INFO [loop_until]: OK (rc = 0) 19:16:49 DEBUG --- stdout --- 19:16:49 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:49 DEBUG --- stderr --- 19:16:49 DEBUG 19:16:49 INFO 19:16:49 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:49 INFO [loop_until]: OK (rc = 0) 19:16:49 DEBUG --- stdout --- 19:16:49 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:49 DEBUG --- stderr --- 19:16:49 DEBUG 19:16:49 INFO 19:16:49 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:16:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:49 INFO [loop_until]: OK (rc = 0) 19:16:49 DEBUG --- stdout --- 19:16:49 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:16:49 DEBUG --- stderr --- 19:16:49 DEBUG 19:16:49 INFO Initializing monitoring instance threads 19:16:49 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 19:16:49 INFO Starting instance threads 19:16:49 INFO 19:16:49 INFO Thread started 19:16:49 INFO [loop_until]: kubectl --namespace=xlou top node 19:16:49 INFO 19:16:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:49 INFO Thread started 19:16:49 INFO [loop_until]: kubectl --namespace=xlou top pods 19:16:49 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377809" 19:16:49 INFO Thread started 19:16:49 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377809" 19:16:50 INFO Thread started 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 INFO Thread started 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 INFO Thread started 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713377810" 19:16:50 INFO Thread started 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713377810" 19:16:50 INFO Thread started 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377810" 19:16:50 INFO Thread started 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377810" 19:16:50 INFO Thread started 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377810" 19:16:50 INFO Thread started 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 INFO Thread started 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 INFO Thread started 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 INFO Thread started 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 DEBUG --- status code --- 19:16:50 INFO Thread started 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG --- http response --- 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:50 DEBUG --- status code --- 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 INFO Thread started 19:16:50 DEBUG --- status code --- 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG ----- output ----- 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 DEBUG --- status code --- 19:16:50 INFO Thread started 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG --- status code --- 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG { 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 DEBUG --- status code --- 19:16:50 INFO Thread started 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 INFO Thread started 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG "data": { 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 INFO Thread started 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:50 DEBUG { 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG { 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG --- http response --- 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 DEBUG ----- output ----- 19:16:50 INFO Thread started 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:50 DEBUG "result": [ 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG { 19:16:50 DEBUG { 19:16:50 DEBUG { 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG ----- output ----- 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 INFO Thread started 19:16:50 DEBUG { 19:16:50 DEBUG { 19:16:50 DEBUG { 19:16:50 DEBUG "data": { 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG "data": { 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG { 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:50 DEBUG ----- output ----- 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG { 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG { 19:16:50 DEBUG "status": "success", 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 DEBUG "metric": { 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG "data": { 19:16:50 DEBUG "data": { 19:16:50 DEBUG "data": { 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG { 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG { 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG --- http response --- 19:16:50 INFO Thread started 19:16:50 DEBUG "data": { 19:16:50 DEBUG "resultType": "vector", 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG "data": { 19:16:50 DEBUG "data": { 19:16:50 DEBUG "result": [ 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG "result": [ 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG "data": { 19:16:50 DEBUG { 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG "data": { 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:16:50 DEBUG "result": [ 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 INFO Thread started 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG "resultType": "vector", 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG { 19:16:50 DEBUG "result": [ 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG "result": [ 19:16:50 DEBUG "result": [ 19:16:50 DEBUG { 19:16:50 DEBUG "data": { 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG "data": { 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG { 19:16:50 DEBUG { 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG "result": [ 19:16:50 DEBUG { 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG "result": [ 19:16:50 DEBUG "metric": { 19:16:50 DEBUG { 19:16:50 DEBUG { 19:16:50 DEBUG { 19:16:50 DEBUG { 19:16:50 DEBUG "metric": { 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG "result": [ 19:16:50 DEBUG "data": { 19:16:50 DEBUG "resultType": "vector", 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:16:50 DEBUG "result": [ 19:16:50 DEBUG }, 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG "metric": { 19:16:50 DEBUG --- http response --- 19:16:50 INFO Thread started 19:16:50 DEBUG "metric": { 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG { 19:16:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:16:50 DEBUG "metric": { 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG "metric": { 19:16:50 DEBUG "metric": { 19:16:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:16:50 DEBUG "result": [ 19:16:50 DEBUG { 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG "result": [ 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG { 19:16:50 DEBUG "value": [ 19:16:50 DEBUG "data": { 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713377810" 19:16:50 DEBUG "instance": "10.142.0.107:9100" 19:16:50 DEBUG }, 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG "metric": { 19:16:50 DEBUG }, 19:16:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:16:50 DEBUG "data": { 19:16:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:16:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:16:50 DEBUG }, 19:16:50 DEBUG { 19:16:50 DEBUG "metric": { 19:16:50 DEBUG "result": [ 19:16:50 DEBUG { 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG "metric": { 19:16:50 DEBUG 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG }, 19:16:50 DEBUG ----- output ----- 19:16:50 INFO Thread started 19:16:50 DEBUG "value": [ 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:16:50 DEBUG "instance": "10.142.0.107:9100" 19:16:50 DEBUG "value": [ 19:16:50 DEBUG }, 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG }, 19:16:50 DEBUG }, 19:16:50 DEBUG "value": [ 19:16:50 DEBUG "metric": { 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG "pod": "ds-cts-0" 19:16:50 DEBUG }, 19:16:50 DEBUG "value": [ 19:16:50 DEBUG --- http response --- 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:16:50 DEBUG 19:16:50 DEBUG "result": [ 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG { 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377810" 19:16:50 DEBUG 19:16:50 INFO Thread started 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG }, 19:16:50 DEBUG 19:16:50 DEBUG "value": [ 19:16:50 DEBUG "result": [ 19:16:50 DEBUG "value": [ 19:16:50 DEBUG "value": [ 19:16:50 DEBUG 19:16:50 DEBUG "pod": "ds-idrepo-0" 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG "metric": { 19:16:50 DEBUG { 19:16:50 DEBUG 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG }, 19:16:50 DEBUG 19:16:50 DEBUG { 19:16:50 DEBUG "value": [ 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:50 DEBUG 19:16:50 DEBUG { 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377810" 19:16:50 INFO Thread started 19:16:50 DEBUG "value": [ 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG { 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG }, 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG "pod": "ds-idrepo-0" 19:16:50 DEBUG "metric": { 19:16:50 DEBUG 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG { 19:16:50 ERROR [http_cmd]: ERROR 19:16:50 DEBUG "value": [ 19:16:50 DEBUG [...] 19:16:50 DEBUG "metric": { 19:16:50 DEBUG 19:16:50 DEBUG "data": { 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG 19:16:50 DEBUG "status": "success", 19:16:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377810" 19:16:50 DEBUG 19:16:50 INFO Thread started 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG "metric": { 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG "value": [ 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG }, 19:16:50 DEBUG "value": [ 19:16:50 DEBUG 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG "status": "success", 19:16:50 ERROR --- status code --- 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:16:50 DEBUG 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG { 19:16:50 DEBUG [...] 19:16:50 DEBUG "data": { 19:16:50 DEBUG 19:16:50 INFO All threads has been started 127.0.0.1 - - [17/Apr/2024 19:16:50] "GET /monitoring/start HTTP/1.1" 200 - 19:16:50 DEBUG [...] 19:16:50 DEBUG 19:16:50 DEBUG "pod": "ds-idrepo-0" 19:16:50 DEBUG 19:16:50 DEBUG [...] 19:16:50 DEBUG [...] 19:16:50 DEBUG 19:16:50 DEBUG "instance": "10.142.0.107:9100" 19:16:50 DEBUG 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG [...] 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG "data": { 19:16:50 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG }, 19:16:50 DEBUG 19:16:50 DEBUG "result": [ 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG [...] 19:16:50 DEBUG }, 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG }, 19:16:50 DEBUG 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:16:50 DEBUG "resultType": "vector", 19:16:50 ERROR --- http response --- 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG "value": [ 19:16:50 DEBUG [...] 19:16:50 DEBUG { 19:16:50 DEBUG "data": { 19:16:50 DEBUG 19:16:50 DEBUG "result": [ 19:16:50 DEBUG [...] 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG "value": [ 19:16:50 DEBUG [...] 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG "value": [ 19:16:50 DEBUG 19:16:50 DEBUG { 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG "result": [ 19:16:50 DEBUG [...] 19:16:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:16:50 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG "metric": { 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG 19:16:50 DEBUG { 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG [...] 19:16:50 DEBUG 19:16:50 DEBUG [...] 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG "data": { 19:16:50 DEBUG 19:16:50 DEBUG { 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:50 DEBUG { 19:16:50 DEBUG 19:16:50 DEBUG }, 19:16:50 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:16:50 DEBUG "result": [ 19:16:50 DEBUG "instance": "10.142.0.43:9100" 19:16:50 DEBUG "metric": { 19:16:50 DEBUG 19:16:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:16:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:16:50 DEBUG }, 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 INFO [http_cmd]: http status code OK 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG "metric": { 19:16:50 DEBUG 19:16:50 DEBUG "value": [ 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG }, 19:16:50 DEBUG { 19:16:50 DEBUG }, 19:16:50 DEBUG "pod": "ds-cts-0" 19:16:50 DEBUG 19:16:50 DEBUG }, 19:16:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:16:50 DEBUG 19:16:50 DEBUG [...] 19:16:50 DEBUG }, 19:16:50 DEBUG 19:16:50 DEBUG "value": [ 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG --- status code --- 19:16:50 DEBUG "result": [ 19:16:50 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:16:50 DEBUG "pod": "ds-cts-2" 19:16:50 DEBUG "data": { 19:16:50 DEBUG { 19:16:50 DEBUG "pod": "ds-cts-0" 19:16:50 DEBUG 19:16:50 DEBUG 1713377809, 19:16:50 DEBUG [...] 19:16:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:16:50 DEBUG "value": [ 19:16:50 DEBUG "metric": { 19:16:50 DEBUG "value": [ 19:16:50 DEBUG }, 19:16:50 DEBUG "instance": "10.142.0.43:9100" 19:16:50 DEBUG "value": [ 19:16:50 DEBUG }, 19:16:50 DEBUG 19:16:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:16:50 DEBUG "value": [ 19:16:50 DEBUG 19:16:50 DEBUG 1713377809, 19:16:50 DEBUG [...] 19:16:50 DEBUG 19:16:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:50 DEBUG { 19:16:50 DEBUG }, 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG }, 19:16:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:16:50 DEBUG "0.0006266223356039801" 19:16:50 DEBUG 19:16:50 DEBUG }, 19:16:50 DEBUG 19:16:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG "value": [ 19:16:50 DEBUG }, 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG "value": [ 19:16:50 DEBUG 19:16:50 DEBUG }, 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG 19:16:50 DEBUG "0" 19:16:50 DEBUG 19:16:50 DEBUG "pod": "ds-idrepo-2" 19:16:50 DEBUG --- http response --- 19:16:50 DEBUG "metric": { 19:16:50 DEBUG "value": [ 19:16:50 DEBUG "result": [ 19:16:50 DEBUG "data": { 19:16:50 DEBUG "value": [ 19:16:50 DEBUG 19:16:50 DEBUG ] 19:16:50 DEBUG 19:16:50 DEBUG "value": [ 19:16:50 DEBUG 19:16:50 DEBUG }, 19:16:50 DEBUG "13628.133333333333" 19:16:50 DEBUG 19:16:50 DEBUG "value": [ 19:16:50 DEBUG "0" 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG "value": [ 19:16:50 DEBUG "pod": "ds-idrepo-2" 19:16:50 DEBUG ] 19:16:50 DEBUG 19:16:50 DEBUG }, 19:16:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:16:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG { 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG [...] 19:16:50 DEBUG }, 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG 19:16:50 DEBUG "value": [ 19:16:50 DEBUG ] 19:16:50 DEBUG 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG ] 19:16:50 DEBUG "0" 19:16:50 DEBUG "1158246400" 19:16:50 DEBUG "pod": "ds-idrepo-2" 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG }, 19:16:50 DEBUG } 19:16:50 DEBUG 19:16:50 DEBUG "value": [ 19:16:50 DEBUG }, 19:16:50 DEBUG ----- output ----- 19:16:50 DEBUG "0" 19:16:50 DEBUG "metric": { 19:16:50 DEBUG "result": [ 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG } 19:16:50 DEBUG "0" 19:16:50 DEBUG [...] 19:16:50 DEBUG 19:16:50 DEBUG } 19:16:50 DEBUG 19:16:50 DEBUG "20385.966666666667" 19:16:50 DEBUG } 19:16:50 DEBUG ] 19:16:50 DEBUG ] 19:16:50 DEBUG }, 19:16:50 DEBUG "0" 19:16:50 DEBUG "value": [ 19:16:50 DEBUG ] 19:16:50 DEBUG "instance": "10.142.0.43:9100" 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG "value": [ 19:16:50 DEBUG { 19:16:50 DEBUG ] 19:16:50 DEBUG "pod": "ds-idrepo-0" 19:16:50 DEBUG { 19:16:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:16:50 DEBUG "value": [ 19:16:50 DEBUG 19:16:50 DEBUG ] 19:16:50 DEBUG ] 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG ] 19:16:50 DEBUG [...] 19:16:50 DEBUG ] 19:16:50 DEBUG ] 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG "value": [ 19:16:50 DEBUG ] 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG } 19:16:50 DEBUG }, 19:16:50 DEBUG "0" 19:16:50 DEBUG ] 19:16:50 DEBUG "status": "success", 19:16:50 DEBUG } 19:16:50 DEBUG }, 19:16:50 DEBUG "metric": { 19:16:50 DEBUG }, 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG "pod": "ds-cts-2" 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG } 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG } 19:16:50 DEBUG ] 19:16:50 DEBUG ] 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG } 19:16:50 DEBUG "0" 19:16:50 DEBUG } 19:16:50 DEBUG "value": [ 19:16:50 DEBUG } 19:16:50 DEBUG 19:16:50 DEBUG "data": { 19:16:50 DEBUG ] 19:16:50 DEBUG "value": [ 19:16:50 DEBUG "instance": "10.142.0.107:9100" 19:16:50 DEBUG "value": [ 19:16:50 DEBUG "0.022692271428890035" 19:16:50 DEBUG }, 19:16:50 DEBUG } 19:16:50 DEBUG ] 19:16:50 DEBUG } 19:16:50 DEBUG [...] 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG "0" 19:16:50 DEBUG ] 19:16:50 DEBUG ] 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG ] 19:16:50 DEBUG 19:16:50 DEBUG "resultType": "vector", 19:16:50 DEBUG } 19:16:50 DEBUG 19:16:50 DEBUG }, 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG ] 19:16:50 DEBUG "value": [ 19:16:50 DEBUG 19:16:50 DEBUG } 19:16:50 DEBUG 19:16:50 DEBUG ] 19:16:50 DEBUG "pod": "ds-cts-2" 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG ] 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG "0.00033333333333333365" 19:16:50 DEBUG } 19:16:50 DEBUG 19:16:50 DEBUG "result": [ 19:16:50 DEBUG } 19:16:50 DEBUG 19:16:50 DEBUG "value": [ 19:16:50 DEBUG "0" 19:16:50 DEBUG } 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:16:50 DEBUG 19:16:50 DEBUG } 19:16:50 DEBUG }, 19:16:50 DEBUG ] 19:16:50 DEBUG } 19:16:50 DEBUG ] 19:16:50 DEBUG ] 19:16:50 DEBUG } 19:16:50 DEBUG [...] 19:16:50 DEBUG { 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG ] 19:16:50 DEBUG ] 19:16:50 DEBUG "0" 19:16:50 DEBUG }, 19:16:50 DEBUG 19:16:50 DEBUG } 19:16:50 DEBUG "value": [ 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG 19:16:50 DEBUG "metric": { 19:16:50 DEBUG [...] 19:16:50 DEBUG 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG ] 19:16:50 DEBUG "value": [ 19:16:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG ] 19:16:50 DEBUG 19:16:50 DEBUG "instance": "10.142.0.107:9100" 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG [...] 19:16:50 DEBUG ] 19:16:50 DEBUG } 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG }, 19:16:50 DEBUG "0" 19:16:50 DEBUG } 19:16:50 DEBUG 19:16:50 DEBUG }, 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG "0" 19:16:50 DEBUG "value": [ 19:16:50 DEBUG ] 19:16:50 DEBUG } 19:16:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:16:50 DEBUG "value": [ 19:16:50 DEBUG 19:16:50 DEBUG 19:16:50 DEBUG ] 19:16:50 DEBUG ] 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG } 19:16:50 DEBUG ] 19:16:50 DEBUG }, 19:16:50 DEBUG 19:16:50 DEBUG "pod": "ds-cts-2" 19:16:50 DEBUG 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG "1" 19:16:50 DEBUG } 19:16:50 DEBUG "value": [ 19:16:50 DEBUG 19:16:50 DEBUG }, 19:16:50 DEBUG "instance": "10.142.0.43:9100" 19:16:50 DEBUG } 19:16:50 DEBUG ] 19:16:50 DEBUG ] 19:16:50 DEBUG } 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG 19:16:50 DEBUG "value": [ 19:16:50 DEBUG }, 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG "0.005" 19:16:50 DEBUG [...] 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG "value": [ 19:16:50 DEBUG } 19:16:50 DEBUG ] 19:16:50 DEBUG ] 19:16:50 DEBUG 19:16:50 DEBUG "0" 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG 19:16:50 DEBUG ] 19:16:50 DEBUG "30220469248" 19:16:50 DEBUG } 19:16:50 DEBUG ] 19:16:50 DEBUG 19:16:50 DEBUG } 19:16:50 DEBUG ] 19:16:50 DEBUG } 19:16:50 DEBUG "instance": "10.142.0.43:9100" 19:16:50 DEBUG ] 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG }, 19:16:50 DEBUG } 19:16:50 DEBUG ] 19:16:50 DEBUG "value": [ 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 DEBUG 1713377810, 19:16:50 DEBUG } 19:16:50 DEBUG "0.0009333333333333489" 19:16:50 DEBUG ] 19:16:50 DEBUG } 19:16:50 DEBUG ] 19:16:50 DEBUG } 19:16:50 DEBUG } 19:16:50 INFO [loop_until]: OK (rc = 0) 19:16:50 DEBUG --- stdout --- 19:16:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 7m 1045Mi am-6844b8c4b7-n64np 8m 3210Mi am-6844b8c4b7-pzj6t 4m 1119Mi ds-cts-0 7m 516Mi ds-cts-1 5m 502Mi ds-cts-2 6m 555Mi ds-idrepo-0 10m 4070Mi ds-idrepo-1 8m 4355Mi ds-idrepo-2 9m 4130Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 947Mi idm-b85f9d978-wnhxb 4m 979Mi lodemon-9886f5b6-dt5fm 1m 55Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 1m 1013Mi 19:16:50 DEBUG --- stderr --- 19:16:50 DEBUG 19:16:50 INFO [loop_until]: OK (rc = 0) 19:16:50 DEBUG --- stdout --- 19:16:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 89m 0% 5498Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 64m 0% 6515Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 60m 0% 5748Mi 5% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 4045Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 62m 0% 2582Mi 2% gke-xlou-cdm-default-pool-1b0769c8-wpwl 155m 0% 4930Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 68m 0% 6403Mi 5% gke-xlou-cdm-ds-0f601751-cf7b 69m 0% 3812Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 64m 0% 5683Mi 5% gke-xlou-cdm-ds-0f601751-j488 50m 0% 1672Mi 1% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 3685Mi 3% gke-xlou-cdm-ds-0f601751-nj6q 81m 0% 2173Mi 1% gke-xlou-cdm-ds-0f601751-nmxm 57m 0% 2091Mi 1% gke-xlou-cdm-frontend-a8771548-i82n 70m 0% 4788Mi 8% 19:16:50 DEBUG --- stderr --- 19:16:50 DEBUG 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713377870" 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713377870" 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377870" 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:17:50 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG --- status code --- 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG ----- output ----- 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG { 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG { 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG "data": { 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG "data": { 19:17:50 DEBUG --- http response --- 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG "result": [ 19:17:50 DEBUG { 19:17:50 DEBUG { 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG "result": [ 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG { 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG { 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG { 19:17:50 DEBUG { 19:17:50 DEBUG --- http response --- 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713377870" 19:17:50 DEBUG "metric": { 19:17:50 DEBUG "data": { 19:17:50 DEBUG "data": { 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:50 DEBUG "metric": { 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:50 DEBUG "pod": "ds-idrepo-0" 19:17:50 DEBUG }, 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG "data": { 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG "data": { 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "result": [ 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG { 19:17:50 DEBUG { 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG { 19:17:50 DEBUG "result": [ 19:17:50 DEBUG 19:17:50 DEBUG { 19:17:50 DEBUG "result": [ 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG }, 19:17:50 DEBUG "data": { 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG "result": [ 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG { 19:17:50 DEBUG 19:17:50 DEBUG "metric": { 19:17:50 DEBUG { 19:17:50 DEBUG "data": { 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:50 DEBUG { 19:17:50 DEBUG "data": { 19:17:50 DEBUG "metric": { 19:17:50 DEBUG 19:17:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:17:50 DEBUG "metric": { 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG 19:17:50 DEBUG "result": [ 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG "metric": { 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG "pod": "ds-cts-0" 19:17:50 DEBUG [...] 19:17:50 DEBUG }, 19:17:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:17:50 DEBUG "result": [ 19:17:50 DEBUG 19:17:50 DEBUG { 19:17:50 DEBUG { 19:17:50 DEBUG { 19:17:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:17:50 DEBUG "result": [ 19:17:50 DEBUG { 19:17:50 DEBUG 19:17:50 DEBUG "value": [ 19:17:50 DEBUG }, 19:17:50 DEBUG { 19:17:50 DEBUG 19:17:50 DEBUG "metric": { 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG }, 19:17:50 DEBUG }, 19:17:50 DEBUG "metric": { 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "metric": { 19:17:50 DEBUG [...] 19:17:50 DEBUG "instance": "10.142.0.107:9100" 19:17:50 DEBUG "data": { 19:17:50 DEBUG "data": { 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG 19:17:50 DEBUG }, 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG }, 19:17:50 DEBUG "pod": "ds-cts-2" 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG }, 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "result": [ 19:17:50 DEBUG "result": [ 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG "value": [ 19:17:50 DEBUG }, 19:17:50 DEBUG [...] 19:17:50 DEBUG 19:17:50 DEBUG "value": [ 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG { 19:17:50 DEBUG { 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG [...] 19:17:50 DEBUG 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:17:50 DEBUG 19:17:50 DEBUG "metric": { 19:17:50 DEBUG "metric": { 19:17:50 DEBUG [...] 19:17:50 DEBUG [...] 19:17:50 DEBUG 19:17:50 DEBUG "0" 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG }, 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:17:50 DEBUG 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 DEBUG "pod": "ds-idrepo-0" 19:17:50 DEBUG "pod": "ds-idrepo-0" 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG ] 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377870" 19:17:50 DEBUG "value": [ 19:17:50 ERROR [http_cmd]: ERROR 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG [...] 19:17:50 DEBUG }, 19:17:50 DEBUG "value": [ 19:17:50 DEBUG }, 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 DEBUG 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 DEBUG [...] 19:17:50 DEBUG } 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG [...] 19:17:50 DEBUG 1713377870, 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 ERROR --- status code --- 19:17:50 DEBUG { 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 19:17:50 DEBUG "pod": "ds-cts-2" 19:17:50 DEBUG }, 19:17:50 DEBUG 19:17:50 DEBUG ] 19:17:50 DEBUG "pod": "ds-cts-0" 19:17:50 DEBUG }, 19:17:50 DEBUG 19:17:50 DEBUG "0" 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377870" 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377870" 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 19:17:50 DEBUG } 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:17:50 DEBUG 19:17:50 DEBUG ] 19:17:50 ERROR --- http response --- 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713377870" 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377870" 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG "data": { 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG 19:17:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG } 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG "0" 19:17:50 DEBUG 19:17:50 DEBUG } 19:17:50 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG --- status code --- 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG "instance": "10.142.0.43:9100" 19:17:50 DEBUG }, 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG [...] 19:17:50 DEBUG 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG "0" 19:17:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG }, 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG "result": [ 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG }, 19:17:50 DEBUG "value": [ 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG 19:17:50 DEBUG [...] 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG ] 19:17:50 DEBUG }, 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "pod": "ds-idrepo-1" 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG { 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG "2.8759451930283646" 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG } 19:17:50 DEBUG "value": [ 19:17:50 DEBUG --- http response --- 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG }, 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:17:50 DEBUG --- http response --- 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 INFO [http_cmd]: http status code OK 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG "metric": { 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG ] 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG ] 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:17:50 DEBUG "7757.5" 19:17:50 DEBUG "value": [ 19:17:50 DEBUG } 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG --- status code --- 19:17:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG "30429568000" 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:50 DEBUG } 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG "pod": "ds-idrepo-2" 19:17:50 DEBUG 19:17:50 DEBUG } 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG "1626087424" 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG { 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG { 19:17:50 DEBUG { 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:50 DEBUG }, 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG { 19:17:50 DEBUG ] 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG ] 19:17:50 DEBUG { 19:17:50 DEBUG }, 19:17:50 DEBUG "pod": "ds-idrepo-2" 19:17:50 DEBUG } 19:17:50 DEBUG { 19:17:50 DEBUG ] 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG ] 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG "4813.772512149415" 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG { 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG --- http response --- 19:17:50 DEBUG "value": [ 19:17:50 DEBUG { 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG } 19:17:50 DEBUG { 19:17:50 DEBUG } 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG "value": [ 19:17:50 DEBUG }, 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG } 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:50 DEBUG } 19:17:50 DEBUG "data": { 19:17:50 DEBUG ] 19:17:50 DEBUG "data": { 19:17:50 DEBUG "data": { 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:50 DEBUG 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG "data": { 19:17:50 DEBUG ] 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG } 19:17:50 DEBUG "data": { 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "data": { 19:17:50 DEBUG ] 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG ] 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG } 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG "data": { 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG ----- output ----- 19:17:50 DEBUG 19:17:50 DEBUG "data": { 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG } 19:17:50 DEBUG "data": { 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG "0" 19:17:50 DEBUG ] 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG } 19:17:50 DEBUG { 19:17:50 DEBUG } 19:17:50 DEBUG "result": [ 19:17:50 DEBUG ] 19:17:50 DEBUG "result": [ 19:17:50 DEBUG { 19:17:50 DEBUG "result": [ 19:17:50 DEBUG { 19:17:50 DEBUG { 19:17:50 DEBUG { 19:17:50 DEBUG 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG "result": [ 19:17:50 DEBUG } 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG "result": [ 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG "0" 19:17:50 DEBUG "result": [ 19:17:50 DEBUG } 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG } 19:17:50 DEBUG { 19:17:50 DEBUG } 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG "metric": { 19:17:50 DEBUG { 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG "status": "success", 19:17:50 DEBUG [...] 19:17:50 DEBUG "result": [ 19:17:50 DEBUG { 19:17:50 DEBUG "result": [ 19:17:50 DEBUG { 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG { 19:17:50 DEBUG "data": { 19:17:50 DEBUG "metric": { 19:17:50 DEBUG } 19:17:50 DEBUG "result": [ 19:17:50 DEBUG "instance": "10.142.0.107:9100" 19:17:50 DEBUG "metric": { 19:17:50 DEBUG "data": { 19:17:50 DEBUG "data": { 19:17:50 DEBUG "data": { 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG 19:17:50 DEBUG "metric": { 19:17:50 DEBUG { 19:17:50 DEBUG "metric": { 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG "metric": { 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG "pod": "ds-cts-0" 19:17:50 DEBUG { 19:17:50 DEBUG }, 19:17:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG "resultType": "vector", 19:17:50 DEBUG { 19:17:50 DEBUG "result": [ 19:17:50 DEBUG { 19:17:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:17:50 DEBUG "metric": { 19:17:50 DEBUG "pod": "ds-cts-0" 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:17:50 DEBUG "result": [ 19:17:50 DEBUG }, 19:17:50 DEBUG "metric": { 19:17:50 DEBUG "value": [ 19:17:50 DEBUG }, 19:17:50 DEBUG "result": [ 19:17:50 DEBUG "result": [ 19:17:50 DEBUG "metric": { 19:17:50 DEBUG 19:17:50 DEBUG "metric": { 19:17:50 DEBUG }, 19:17:50 DEBUG "pod": "ds-idrepo-0" 19:17:50 DEBUG }, 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG }, 19:17:50 DEBUG { 19:17:50 DEBUG "metric": { 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG "value": [ 19:17:50 DEBUG { 19:17:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:17:50 DEBUG 19:17:50 DEBUG "instance": "10.142.0.107:9100" 19:17:50 DEBUG "value": [ 19:17:50 DEBUG }, 19:17:50 DEBUG "value": [ 19:17:50 DEBUG } 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "instance": "10.142.0.107:9100" 19:17:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:17:50 DEBUG 19:17:50 DEBUG { 19:17:50 DEBUG "metric": { 19:17:50 DEBUG 19:17:50 DEBUG "metric": { 19:17:50 DEBUG }, 19:17:50 DEBUG "pod": "ds-idrepo-1" 19:17:50 DEBUG }, 19:17:50 DEBUG 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG }, 19:17:50 DEBUG }, 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 19:17:50 DEBUG "instance": "10.142.0.107:9100" 19:17:50 DEBUG [...] 19:17:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:17:50 DEBUG "value": [ 19:17:50 DEBUG }, 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG }, 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG [...] 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG }, 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG 19:17:50 DEBUG [...] 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG [...] 19:17:50 DEBUG 19:17:50 DEBUG [...] 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG [...] 19:17:50 DEBUG "0" 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG [...] 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG "pod": "ds-cts-2" 19:17:50 DEBUG 19:17:50 DEBUG [...] 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG ] 19:17:50 DEBUG [...] 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG [...] 19:17:50 DEBUG 19:17:50 DEBUG }, 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG [...] 19:17:50 DEBUG "instance": "10.142.0.43:9100" 19:17:50 DEBUG 19:17:50 DEBUG } 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG [...] 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG }, 19:17:50 DEBUG 19:17:50 DEBUG ] 19:17:50 DEBUG 19:17:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG "pod": "ds-cts-2" 19:17:50 DEBUG 19:17:50 DEBUG "pod": "ds-cts-0" 19:17:50 DEBUG [...] 19:17:50 DEBUG 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "instance": "10.142.0.43:9100" 19:17:50 DEBUG } 19:17:50 DEBUG 19:17:50 DEBUG }, 19:17:50 DEBUG "pod": "ds-idrepo-2" 19:17:50 DEBUG 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG 19:17:50 DEBUG }, 19:17:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:17:50 DEBUG }, 19:17:50 DEBUG 19:17:50 DEBUG 19:17:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:17:50 DEBUG }, 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "instance": "10.142.0.43:9100" 19:17:50 DEBUG }, 19:17:50 DEBUG }, 19:17:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:17:50 DEBUG "0" 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG }, 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG }, 19:17:50 DEBUG } 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "value": [ 19:17:50 DEBUG }, 19:17:50 DEBUG "instance": "10.142.0.43:9100" 19:17:50 DEBUG "value": [ 19:17:50 DEBUG ] 19:17:50 DEBUG "value": [ 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG 19:17:50 DEBUG "16092383.953378333" 19:17:50 DEBUG "value": [ 19:17:50 DEBUG "0.023600000000000187" 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG "value": [ 19:17:50 DEBUG }, 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG "0" 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG "0" 19:17:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:17:50 DEBUG ] 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG ] 19:17:50 DEBUG "0" 19:17:50 DEBUG "14970960.589189999" 19:17:50 DEBUG "0" 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG "value": [ 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG "0.005" 19:17:50 DEBUG ] 19:17:50 DEBUG }, 19:17:50 DEBUG } 19:17:50 DEBUG "523.8666666666667" 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG ] 19:17:50 DEBUG ] 19:17:50 DEBUG "413.36967833333335" 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG "value": [ 19:17:50 DEBUG ] 19:17:50 DEBUG ] 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG "0.016666666666666677" 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG 1713377870, 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG "412.1514705555556" 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG ] 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 DEBUG } 19:17:50 INFO 19:17:50 INFO [loop_until]: kubectl --namespace=xlou top pods 19:17:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:17:50 INFO [loop_until]: OK (rc = 0) 19:17:50 DEBUG --- stdout --- 19:17:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 3825m 3880Mi am-6844b8c4b7-n64np 1004m 3315Mi am-6844b8c4b7-pzj6t 2078m 1321Mi ds-cts-0 389m 563Mi ds-cts-1 95m 522Mi ds-cts-2 181m 581Mi ds-idrepo-0 270m 4081Mi ds-idrepo-1 33m 4356Mi ds-idrepo-2 79m 4131Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 946Mi idm-b85f9d978-wnhxb 4m 979Mi lodemon-9886f5b6-dt5fm 113m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 2879m 1459Mi 19:17:50 DEBUG --- stderr --- 19:17:50 DEBUG 19:17:50 INFO 19:17:50 INFO [loop_until]: kubectl --namespace=xlou top node 19:17:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:17:50 INFO [loop_until]: OK (rc = 0) 19:17:50 DEBUG --- stdout --- 19:17:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 85m 0% 5486Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 179m 0% 6506Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 2106m 7% 5939Mi 5% gke-xlou-cdm-default-pool-1b0769c8-qkns 2327m 7% 4204Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 785m 2% 2680Mi 2% gke-xlou-cdm-default-pool-1b0769c8-wpwl 159m 0% 4932Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1357m 4% 6433Mi 5% gke-xlou-cdm-ds-0f601751-cf7b 1942m 6% 4002Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 102m 0% 5678Mi 5% gke-xlou-cdm-ds-0f601751-j488 52m 0% 1678Mi 1% gke-xlou-cdm-ds-0f601751-mc5d 457m 1% 3583Mi 3% gke-xlou-cdm-ds-0f601751-nj6q 79m 0% 2174Mi 1% gke-xlou-cdm-ds-0f601751-nmxm 441m 1% 2092Mi 1% gke-xlou-cdm-frontend-a8771548-i82n 2825m 17% 5221Mi 8% 19:17:50 DEBUG --- stderr --- 19:17:50 DEBUG 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG { 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "data": { 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "result": [ 19:18:50 DEBUG { 19:18:50 DEBUG "metric": { 19:18:50 DEBUG "pod": "ds-idrepo-0" 19:18:50 DEBUG }, 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG [...] 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG "pod": "ds-cts-2" 19:18:50 DEBUG }, 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG "0" 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG { 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "data": { 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "result": [ 19:18:50 DEBUG { 19:18:50 DEBUG "metric": { 19:18:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:18:50 DEBUG }, 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG [...] 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:18:50 DEBUG }, 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG "1304.6333333333332" 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG { 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "data": { 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "result": [ 19:18:50 DEBUG { 19:18:50 DEBUG "metric": { 19:18:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:18:50 DEBUG }, 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG [...] 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:18:50 DEBUG }, 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG "0" 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG --- http response --- 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG { 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:18:50 DEBUG "data": { 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377930" 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713377930" 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG ----- output ----- 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 DEBUG { 19:18:50 DEBUG "result": [ 19:18:50 DEBUG { 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG { 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "data": { 19:18:50 DEBUG "metric": { 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG "data": { 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "pod": "ds-cts-0" 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "result": [ 19:18:50 DEBUG }, 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG "result": [ 19:18:50 DEBUG { 19:18:50 DEBUG "value": [ 19:18:50 DEBUG { 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:18:50 DEBUG "metric": { 19:18:50 DEBUG 19:18:50 DEBUG "metric": { 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 DEBUG 19:18:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:18:50 DEBUG { 19:18:50 DEBUG }, 19:18:50 DEBUG 19:18:50 DEBUG }, 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "value": [ 19:18:50 DEBUG [...] 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "data": { 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG "result": [ 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG { 19:18:50 DEBUG [...] 19:18:50 DEBUG "pod": "ds-cts-2" 19:18:50 DEBUG [...] 19:18:50 DEBUG "metric": { 19:18:50 DEBUG 19:18:50 DEBUG }, 19:18:50 DEBUG 19:18:50 DEBUG "pod": "ds-cts-0" 19:18:50 DEBUG 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 19:18:50 DEBUG }, 19:18:50 DEBUG 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:18:50 DEBUG "0" 19:18:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:18:50 DEBUG 19:18:50 DEBUG }, 19:18:50 DEBUG ] 19:18:50 DEBUG }, 19:18:50 DEBUG 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 ERROR [http_cmd]: ERROR 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713377930" 19:18:50 DEBUG "value": [ 19:18:50 DEBUG } 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 DEBUG "value": [ 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 DEBUG 19:18:50 DEBUG --- status code --- 19:18:50 ERROR --- status code --- 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG ] 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377930" 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713377930" 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG [...] 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:18:50 DEBUG "0" 19:18:50 DEBUG } 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG "0" 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG 19:18:50 DEBUG --- http response --- 19:18:50 ERROR --- http response --- 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG ] 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377930" 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377930" 19:18:50 DEBUG } 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG ] 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 DEBUG --- status code --- 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377930" 19:18:50 DEBUG } 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377930" 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 19:18:50 DEBUG 19:18:50 DEBUG ----- output ----- 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 DEBUG --- status code --- 19:18:50 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG ] 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG --- http response --- 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG "pod": "ds-cts-2" 19:18:50 DEBUG { 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG } 19:18:50 DEBUG { 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG { 19:18:50 DEBUG }, 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG } 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "data": { 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:50 DEBUG "data": { 19:18:50 DEBUG { 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG "data": { 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG "resultType": "vector", 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 INFO [http_cmd]: http status code OK 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG { 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:18:50 DEBUG "0" 19:18:50 DEBUG "result": [ 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG --- status code --- 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG { 19:18:50 DEBUG { 19:18:50 DEBUG { 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG { 19:18:50 DEBUG "result": [ 19:18:50 DEBUG "data": { 19:18:50 DEBUG "result": [ 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG ] 19:18:50 DEBUG { 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:50 DEBUG { 19:18:50 DEBUG { 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:50 DEBUG { 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "data": { 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG { 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG { 19:18:50 DEBUG { 19:18:50 DEBUG } 19:18:50 DEBUG "metric": { 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG --- http response --- 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "data": { 19:18:50 DEBUG "data": { 19:18:50 DEBUG "data": { 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "data": { 19:18:50 DEBUG "metric": { 19:18:50 DEBUG "result": [ 19:18:50 DEBUG "metric": { 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG ] 19:18:50 DEBUG "instance": "10.142.0.107:9100" 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:50 DEBUG "data": { 19:18:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:50 DEBUG "data": { 19:18:50 DEBUG { 19:18:50 DEBUG { 19:18:50 DEBUG "data": { 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "result": [ 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:18:50 DEBUG { 19:18:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:18:50 DEBUG "data": { 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG ----- output ----- 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "result": [ 19:18:50 DEBUG "result": [ 19:18:50 DEBUG "result": [ 19:18:50 DEBUG { 19:18:50 DEBUG "result": [ 19:18:50 DEBUG }, 19:18:50 DEBUG "metric": { 19:18:50 DEBUG }, 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG }, 19:18:50 DEBUG { 19:18:50 DEBUG { 19:18:50 DEBUG "result": [ 19:18:50 DEBUG { 19:18:50 DEBUG "result": [ 19:18:50 DEBUG "data": { 19:18:50 DEBUG "data": { 19:18:50 DEBUG "result": [ 19:18:50 DEBUG { 19:18:50 DEBUG { 19:18:50 DEBUG { 19:18:50 DEBUG "metric": { 19:18:50 DEBUG { 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:18:50 DEBUG }, 19:18:50 DEBUG "result": [ 19:18:50 DEBUG { 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG { 19:18:50 DEBUG "status": "success", 19:18:50 DEBUG { 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG { 19:18:50 DEBUG "metric": { 19:18:50 DEBUG "metric": { 19:18:50 DEBUG "metric": { 19:18:50 DEBUG "instance": "10.142.0.107:9100" 19:18:50 DEBUG "metric": { 19:18:50 DEBUG 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "metric": { 19:18:50 DEBUG "data": { 19:18:50 DEBUG "data": { 19:18:50 DEBUG "metric": { 19:18:50 DEBUG "data": { 19:18:50 DEBUG "metric": { 19:18:50 DEBUG "result": [ 19:18:50 DEBUG "result": [ 19:18:50 DEBUG "metric": { 19:18:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:18:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:18:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:18:50 DEBUG }, 19:18:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "pod": "ds-cts-0" 19:18:50 DEBUG "resultType": "vector", 19:18:50 DEBUG "pod": "ds-idrepo-0" 19:18:50 DEBUG { 19:18:50 DEBUG { 19:18:50 DEBUG "pod": "ds-idrepo-0" 19:18:50 DEBUG }, 19:18:50 DEBUG }, 19:18:50 DEBUG }, 19:18:50 DEBUG "value": [ 19:18:50 DEBUG }, 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG }, 19:18:50 DEBUG "result": [ 19:18:50 DEBUG "result": [ 19:18:50 DEBUG }, 19:18:50 DEBUG "result": [ 19:18:50 DEBUG }, 19:18:50 DEBUG "metric": { 19:18:50 DEBUG "metric": { 19:18:50 DEBUG }, 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 19:18:50 DEBUG "value": [ 19:18:50 DEBUG [...] 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG "value": [ 19:18:50 DEBUG { 19:18:50 DEBUG { 19:18:50 DEBUG "value": [ 19:18:50 DEBUG { 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "instance": "10.142.0.107:9100" 19:18:50 DEBUG "instance": "10.142.0.107:9100" 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG [...] 19:18:50 DEBUG [...] 19:18:50 DEBUG [...] 19:18:50 DEBUG 19:18:50 DEBUG "metric": { 19:18:50 DEBUG "metric": { 19:18:50 DEBUG 19:18:50 DEBUG "metric": { 19:18:50 DEBUG 19:18:50 DEBUG }, 19:18:50 DEBUG }, 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:18:50 DEBUG "pod": "ds-idrepo-0" 19:18:50 DEBUG 19:18:50 DEBUG "instance": "10.142.0.107:9100" 19:18:50 DEBUG 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG [...] 19:18:50 DEBUG [...] 19:18:50 DEBUG 19:18:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG }, 19:18:50 DEBUG }, 19:18:50 DEBUG 19:18:50 DEBUG }, 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG [...] 19:18:50 DEBUG [...] 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG [...] 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "value": [ 19:18:50 DEBUG [...] 19:18:50 DEBUG "value": [ 19:18:50 DEBUG [...] 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG [...] 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG [...] 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG }, 19:18:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:18:50 DEBUG "pod": "ds-idrepo-2" 19:18:50 DEBUG "instance": "10.142.0.43:9100" 19:18:50 DEBUG }, 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG }, 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG [...] 19:18:50 DEBUG [...] 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:18:50 DEBUG "instance": "10.142.0.43:9100" 19:18:50 DEBUG }, 19:18:50 DEBUG "1304.8" 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:18:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:18:50 DEBUG 19:18:50 DEBUG }, 19:18:50 DEBUG }, 19:18:50 DEBUG "value": [ 19:18:50 DEBUG ] 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG "pod": "ds-idrepo-2" 19:18:50 DEBUG [...] 19:18:50 DEBUG 19:18:50 DEBUG "pod": "ds-cts-2" 19:18:50 DEBUG [...] 19:18:50 DEBUG "pod": "ds-idrepo-2" 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG "pod": "ds-idrepo-2" 19:18:50 DEBUG }, 19:18:50 DEBUG }, 19:18:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG "0" 19:18:50 DEBUG "1315.2506458581317" 19:18:50 DEBUG } 19:18:50 DEBUG }, 19:18:50 DEBUG [...] 19:18:50 DEBUG 19:18:50 DEBUG }, 19:18:50 DEBUG 19:18:50 DEBUG }, 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG }, 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "value": [ 19:18:50 DEBUG }, 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG "5266538496" 19:18:50 DEBUG ] 19:18:50 DEBUG "0.021000000000000067" 19:18:50 DEBUG ] 19:18:50 DEBUG ] 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "instance": "10.142.0.43:9100" 19:18:50 DEBUG "instance": "10.142.0.43:9100" 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG "value": [ 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG 19:18:50 DEBUG "0" 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG }, 19:18:50 DEBUG }, 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG "75680.5" 19:18:50 DEBUG "8.031468374903051" 19:18:50 DEBUG ] 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:18:50 DEBUG }, 19:18:50 DEBUG 19:18:50 DEBUG ] 19:18:50 DEBUG "0" 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "0" 19:18:50 DEBUG ] 19:18:50 DEBUG "31554630656" 19:18:50 DEBUG ] 19:18:50 DEBUG "1305.2666666666667" 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG "0" 19:18:50 DEBUG "pod": "ds-idrepo-2" 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "instance": "10.142.0.43:9100" 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG }, 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG }, 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG "0.02886666666666701" 19:18:50 DEBUG "28978280.8" 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG ] 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG "value": [ 19:18:50 DEBUG "0.005" 19:18:50 DEBUG "value": [ 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG ] 19:18:50 DEBUG ] 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG ] 19:18:50 DEBUG 1713377930, 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG "0" 19:18:50 DEBUG } 19:18:50 DEBUG "28983028.733333334" 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG ] 19:18:50 DEBUG ] 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG ] 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 DEBUG } 19:18:50 INFO 19:18:50 INFO [loop_until]: kubectl --namespace=xlou top pods 19:18:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:18:50 INFO [loop_until]: OK (rc = 0) 19:18:50 DEBUG --- stdout --- 19:18:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 9091m 4979Mi am-6844b8c4b7-n64np 5730m 4307Mi am-6844b8c4b7-pzj6t 4963m 4429Mi ds-cts-0 3221m 1834Mi ds-cts-1 3226m 1831Mi ds-cts-2 3352m 1783Mi ds-idrepo-0 9225m 4075Mi ds-idrepo-1 258m 4360Mi ds-idrepo-2 319m 4132Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 946Mi idm-b85f9d978-wnhxb 4m 979Mi lodemon-9886f5b6-dt5fm 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7956m 4170Mi 19:18:50 DEBUG --- stderr --- 19:18:50 DEBUG 19:18:50 INFO 19:18:50 INFO [loop_until]: kubectl --namespace=xlou top node 19:18:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:18:50 INFO [loop_until]: OK (rc = 0) 19:18:50 DEBUG --- stdout --- 19:18:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 81m 0% 5511Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 310m 1% 6526Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 5983m 20% 6983Mi 6% gke-xlou-cdm-default-pool-1b0769c8-qkns 5215m 17% 7335Mi 6% gke-xlou-cdm-default-pool-1b0769c8-w15q 3264m 10% 3958Mi 3% gke-xlou-cdm-default-pool-1b0769c8-wpwl 151m 0% 4947Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9222m 30% 6414Mi 5% gke-xlou-cdm-ds-0f601751-cf7b 27395m 91% 8772Mi 7% gke-xlou-cdm-ds-0f601751-hmm7 388m 1% 5704Mi 5% gke-xlou-cdm-ds-0f601751-j488 51m 0% 1679Mi 1% gke-xlou-cdm-ds-0f601751-mc5d 3263m 10% 4922Mi 4% gke-xlou-cdm-ds-0f601751-nj6q 81m 0% 2179Mi 1% gke-xlou-cdm-ds-0f601751-nmxm 3359m 11% 3433Mi 3% gke-xlou-cdm-frontend-a8771548-i82n 8191m 51% 8349Mi 14% 19:18:50 DEBUG --- stderr --- 19:18:50 DEBUG 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG { 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "data": { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "result": [ 19:19:50 DEBUG { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "pod": "ds-idrepo-0" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "pod": "ds-cts-2" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG "0" 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG { 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "data": { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "result": [ 19:19:50 DEBUG { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG "0" 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG { 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "data": { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "result": [ 19:19:50 DEBUG { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG "3.9774543922300833" 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG { 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "data": { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "result": [ 19:19:50 DEBUG { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG "1345.0333333333333" 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- http response --- 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713377990" 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:19:50 DEBUG ----- output ----- 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 DEBUG { 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "data": { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "result": [ 19:19:50 DEBUG { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "pod": "ds-cts-0" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "pod": "ds-cts-2" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG "0" 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG { 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "data": { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "result": [ 19:19:50 DEBUG { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "instance": "10.142.0.107:9100" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "instance": "10.142.0.43:9100" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG "32788536320" 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG { 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "data": { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "result": [ 19:19:50 DEBUG { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG "0" 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:50 ERROR [http_cmd]: ERROR 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG ----- output ----- 19:19:50 ERROR --- status code --- 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG { 19:19:50 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG "status": "success", 19:19:50 ERROR --- http response --- 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG "data": { 19:19:50 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:50 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:19:50 DEBUG "result": [ 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG { 19:19:50 DEBUG { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "pod": "ds-idrepo-0" 19:19:50 DEBUG "data": { 19:19:50 DEBUG }, 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "value": [ 19:19:50 DEBUG "result": [ 19:19:50 DEBUG 19:19:50 DEBUG { 19:19:50 DEBUG 19:19:50 DEBUG "metric": { 19:19:50 DEBUG 19:19:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:19:50 DEBUG [...] 19:19:50 DEBUG }, 19:19:50 DEBUG 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "pod": "ds-idrepo-2" 19:19:50 DEBUG 19:19:50 DEBUG }, 19:19:50 DEBUG [...] 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG 19:19:50 DEBUG "0" 19:19:50 DEBUG 19:19:50 DEBUG ] 19:19:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:19:50 DEBUG } 19:19:50 DEBUG }, 19:19:50 DEBUG ] 19:19:50 DEBUG "value": [ 19:19:50 DEBUG } 19:19:50 DEBUG 1713377990, 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 DEBUG } 19:19:50 DEBUG "1324.4333333333334" 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG { 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "data": { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "result": [ 19:19:50 DEBUG { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "pod": "ds-idrepo-0" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "pod": "ds-idrepo-2" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG "0" 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG { 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "data": { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "result": [ 19:19:50 DEBUG { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG "3750095.321116411" 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG { 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "data": { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "result": [ 19:19:50 DEBUG { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "pod": "ds-cts-0" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "pod": "ds-cts-2" 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG "0" 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 DEBUG --- status code --- 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377990" 19:19:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713377990" 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG --- status code --- 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- status code --- 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG --- http response --- 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- http response --- 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG --- http response --- 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG { 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG "data": { 19:19:50 DEBUG { 19:19:50 INFO [http_cmd]: http status code OK 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG { 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:50 DEBUG { 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG { 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG { 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG --- status code --- 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG { 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:50 DEBUG "result": [ 19:19:50 DEBUG "data": { 19:19:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:50 DEBUG { 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "data": { 19:19:50 DEBUG { 19:19:50 DEBUG "data": { 19:19:50 DEBUG "data": { 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG "data": { 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG --- http response --- 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "data": { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG { 19:19:50 DEBUG { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "result": [ 19:19:50 DEBUG "data": { 19:19:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "result": [ 19:19:50 DEBUG "data": { 19:19:50 DEBUG "result": [ 19:19:50 DEBUG "result": [ 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "result": [ 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:19:50 DEBUG { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG ----- output ----- 19:19:50 DEBUG "result": [ 19:19:50 DEBUG { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG { 19:19:50 DEBUG { 19:19:50 DEBUG "data": { 19:19:50 DEBUG { 19:19:50 DEBUG "data": { 19:19:50 DEBUG "data": { 19:19:50 DEBUG }, 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "result": [ 19:19:50 DEBUG { 19:19:50 DEBUG { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "result": [ 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG "value": [ 19:19:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:19:50 DEBUG { 19:19:50 DEBUG "status": "success", 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:19:50 DEBUG { 19:19:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:19:50 DEBUG "instance": "10.142.0.107:9100" 19:19:50 DEBUG "result": [ 19:19:50 DEBUG "pod": "ds-cts-0" 19:19:50 DEBUG "result": [ 19:19:50 DEBUG "result": [ 19:19:50 DEBUG 19:19:50 DEBUG }, 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "data": { 19:19:50 DEBUG "instance": "10.142.0.107:9100" 19:19:50 DEBUG }, 19:19:50 DEBUG "metric": { 19:19:50 DEBUG }, 19:19:50 DEBUG }, 19:19:50 DEBUG { 19:19:50 DEBUG }, 19:19:50 DEBUG { 19:19:50 DEBUG { 19:19:50 DEBUG 19:19:50 DEBUG "value": [ 19:19:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:19:50 DEBUG "resultType": "vector", 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:19:50 DEBUG "value": [ 19:19:50 DEBUG "value": [ 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "value": [ 19:19:50 DEBUG "metric": { 19:19:50 DEBUG "metric": { 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG }, 19:19:50 DEBUG "result": [ 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG }, 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "pod": "ds-idrepo-0" 19:19:50 DEBUG 19:19:50 DEBUG "instance": "10.142.0.107:9100" 19:19:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG "value": [ 19:19:50 DEBUG { 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG }, 19:19:50 DEBUG 19:19:50 DEBUG }, 19:19:50 DEBUG }, 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "metric": { 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG "value": [ 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG "instance": "10.142.0.107:9100" 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG }, 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG }, 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG 19:19:50 DEBUG [...] 19:19:50 DEBUG [...] 19:19:50 DEBUG "value": [ 19:19:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:19:50 DEBUG 19:19:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:19:50 DEBUG "instance": "10.142.0.43:9100" 19:19:50 DEBUG 19:19:50 DEBUG "pod": "ds-cts-2" 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG }, 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "instance": "10.142.0.43:9100" 19:19:50 DEBUG }, 19:19:50 DEBUG 19:19:50 DEBUG }, 19:19:50 DEBUG }, 19:19:50 DEBUG 19:19:50 DEBUG }, 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG "0" 19:19:50 DEBUG "value": [ 19:19:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:19:50 DEBUG [...] 19:19:50 DEBUG }, 19:19:50 DEBUG "value": [ 19:19:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:19:50 DEBUG "value": [ 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 19:19:50 DEBUG ] 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG }, 19:19:50 DEBUG 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG }, 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG "pod": "ds-idrepo-2" 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG "instance": "10.142.0.43:9100" 19:19:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:19:50 DEBUG } 19:19:50 DEBUG "2081084.773504414" 19:19:50 DEBUG "value": [ 19:19:50 DEBUG 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG "8.037639081950312" 19:19:50 DEBUG "value": [ 19:19:50 DEBUG "7107977216" 19:19:50 DEBUG "0.02199999999999997" 19:19:50 DEBUG }, 19:19:50 DEBUG "0.000017330000000000002" 19:19:50 DEBUG }, 19:19:50 DEBUG }, 19:19:50 DEBUG ] 19:19:50 DEBUG ] 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG 19:19:50 DEBUG "0.02960000000000029" 19:19:50 DEBUG ] 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG ] 19:19:50 DEBUG ] 19:19:50 DEBUG "value": [ 19:19:50 DEBUG ] 19:19:50 DEBUG "value": [ 19:19:50 DEBUG "value": [ 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG "154550.5" 19:19:50 DEBUG "instance": "10.142.0.43:9100" 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG "1345.6333333333332" 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG } 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG }, 19:19:50 DEBUG ] 19:19:50 DEBUG ] 19:19:50 DEBUG ] 19:19:50 DEBUG ] 19:19:50 DEBUG "0" 19:19:50 DEBUG ] 19:19:50 DEBUG "28633300.133333333" 19:19:50 DEBUG "0.005" 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG "value": [ 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG 1713377990, 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG ] 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG "28638470.93333333" 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 DEBUG ] 19:19:50 DEBUG } 19:19:50 DEBUG } 19:19:50 INFO 19:19:50 INFO [loop_until]: kubectl --namespace=xlou top pods 19:19:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:19:50 INFO [loop_until]: OK (rc = 0) 19:19:50 DEBUG --- stdout --- 19:19:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 8189m 6231Mi am-6844b8c4b7-n64np 5981m 4831Mi am-6844b8c4b7-pzj6t 5123m 4907Mi ds-cts-0 2995m 2793Mi ds-cts-1 2982m 2623Mi ds-cts-2 3047m 3016Mi ds-idrepo-0 9507m 4112Mi ds-idrepo-1 268m 4360Mi ds-idrepo-2 257m 4133Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 947Mi idm-b85f9d978-wnhxb 4m 979Mi lodemon-9886f5b6-dt5fm 8m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8185m 6415Mi 19:19:50 DEBUG --- stderr --- 19:19:50 DEBUG 19:19:50 INFO 19:19:50 INFO [loop_until]: kubectl --namespace=xlou top node 19:19:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:19:50 INFO [loop_until]: OK (rc = 0) 19:19:50 DEBUG --- stdout --- 19:19:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 80m 0% 5509Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 345m 1% 6526Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6035m 20% 7405Mi 6% gke-xlou-cdm-default-pool-1b0769c8-qkns 5191m 17% 7746Mi 6% gke-xlou-cdm-default-pool-1b0769c8-w15q 3008m 10% 5036Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 136m 0% 4923Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9370m 31% 6437Mi 5% gke-xlou-cdm-ds-0f601751-cf7b 26291m 88% 10029Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 318m 1% 5708Mi 5% gke-xlou-cdm-ds-0f601751-j488 52m 0% 1669Mi 1% gke-xlou-cdm-ds-0f601751-mc5d 2913m 9% 5967Mi 5% gke-xlou-cdm-ds-0f601751-nj6q 79m 0% 2183Mi 1% gke-xlou-cdm-ds-0f601751-nmxm 2939m 9% 4424Mi 3% gke-xlou-cdm-frontend-a8771548-i82n 8211m 51% 9951Mi 16% 19:19:50 DEBUG --- stderr --- 19:19:50 DEBUG 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "1241.6" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "pod": "ds-idrepo-0" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "pod": "ds-cts-2" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "0" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG { 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:20:50 DEBUG 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG 19:20:50 DEBUG { 19:20:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG }, 19:20:50 DEBUG "data": { 19:20:50 DEBUG "value": [ 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "result": [ 19:20:50 DEBUG "0" 19:20:50 DEBUG { 19:20:50 DEBUG ] 19:20:50 DEBUG "metric": { 19:20:50 DEBUG } 19:20:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:20:50 DEBUG ] 19:20:50 DEBUG }, 19:20:50 DEBUG } 19:20:50 DEBUG "value": [ 19:20:50 DEBUG } 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "5576.3049541771825" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378050" 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG { 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG "data": { 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:20:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG }, 19:20:50 DEBUG { 19:20:50 DEBUG "value": [ 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG 19:20:50 DEBUG "data": { 19:20:50 DEBUG 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG 19:20:50 DEBUG "result": [ 19:20:50 DEBUG [...] 19:20:50 DEBUG { 19:20:50 DEBUG 19:20:50 DEBUG "metric": { 19:20:50 DEBUG 19:20:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:20:50 DEBUG 19:20:50 DEBUG }, 19:20:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:20:50 DEBUG "value": [ 19:20:50 DEBUG }, 19:20:50 DEBUG 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG 19:20:50 DEBUG "0" 19:20:50 DEBUG [...] 19:20:50 DEBUG ] 19:20:50 DEBUG 19:20:50 DEBUG } 19:20:50 DEBUG 19:20:50 DEBUG ] 19:20:50 DEBUG 19:20:50 DEBUG } 19:20:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:20:50 DEBUG } 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "0" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "instance": "10.142.0.107:9100" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "instance": "10.142.0.43:9100" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "34028327936" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "1240.9" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713378050" 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "pod": "ds-cts-0" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "pod": "ds-cts-2" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "0" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "pod": "ds-idrepo-0" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "pod": "ds-idrepo-2" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "0" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378050" 19:20:50 DEBUG "metric": { 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 DEBUG "pod": "ds-cts-0" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "pod": "ds-cts-2" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "0" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 ERROR [http_cmd]: ERROR 19:20:50 ERROR --- status code --- 19:20:50 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:20:50 ERROR --- http response --- 19:20:50 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:20:50 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG { 19:20:50 DEBUG { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "instance": "10.142.0.107:9100" 19:20:50 DEBUG "pod": "ds-idrepo-0" 19:20:50 DEBUG }, 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "instance": "10.142.0.43:9100" 19:20:50 DEBUG "pod": "ds-idrepo-2" 19:20:50 DEBUG }, 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "0.03020000000000079" 19:20:50 DEBUG "0" 19:20:50 DEBUG ] 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "pod": "idm-b85f9d978-wnhxb" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "1003.1967138520772" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713378050" 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "pod": "idm-b85f9d978-wnhxb" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "0" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "8353071104" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "225527" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378050" 19:20:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378050" 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "pod": "ds-cts-0" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "pod": "ds-cts-2" 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "0.000017330000000000002" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG ----- output ----- 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG { 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG "data": { 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG "result": [ 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG { 19:20:50 DEBUG { 19:20:50 DEBUG { 19:20:50 DEBUG { 19:20:50 DEBUG { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG "data": { 19:20:50 DEBUG "data": { 19:20:50 DEBUG "data": { 19:20:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:20:50 DEBUG "data": { 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG }, 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG "result": [ 19:20:50 DEBUG "result": [ 19:20:50 DEBUG "result": [ 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG "value": [ 19:20:50 DEBUG "result": [ 19:20:50 DEBUG { 19:20:50 DEBUG { 19:20:50 DEBUG { 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG 19:20:50 DEBUG { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "metric": { 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "pod": "ds-idrepo-0" 19:20:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:20:50 DEBUG "instance": "10.142.0.107:9100" 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG 19:20:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:20:50 DEBUG }, 19:20:50 DEBUG }, 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:20:50 DEBUG }, 19:20:50 INFO [http_cmd]: http status code OK 19:20:50 DEBUG "value": [ 19:20:50 DEBUG "value": [ 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG "value": [ 19:20:50 DEBUG --- status code --- 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG { 19:20:50 DEBUG 19:20:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG 19:20:50 DEBUG --- http response --- 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG "data": { 19:20:50 DEBUG 19:20:50 DEBUG [...] 19:20:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:20:50 DEBUG [...] 19:20:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:20:50 DEBUG 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG [...] 19:20:50 DEBUG 19:20:50 DEBUG ----- output ----- 19:20:50 DEBUG 19:20:50 DEBUG }, 19:20:50 DEBUG 19:20:50 DEBUG "result": [ 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG { 19:20:50 DEBUG 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG { 19:20:50 DEBUG 19:20:50 DEBUG 19:20:50 DEBUG "status": "success", 19:20:50 DEBUG 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "instance": "10.142.0.43:9100" 19:20:50 DEBUG "metric": { 19:20:50 DEBUG 19:20:50 DEBUG "pod": "ds-idrepo-2" 19:20:50 DEBUG "data": { 19:20:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:20:50 DEBUG "8.028381036983399" 19:20:50 DEBUG }, 19:20:50 DEBUG "instance": "10.142.0.107:9100" 19:20:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:20:50 DEBUG }, 19:20:50 DEBUG "resultType": "vector", 19:20:50 DEBUG }, 19:20:50 DEBUG ] 19:20:50 DEBUG "value": [ 19:20:50 DEBUG }, 19:20:50 DEBUG }, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG "result": [ 19:20:50 DEBUG "value": [ 19:20:50 DEBUG } 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "value": [ 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG { 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG ] 19:20:50 DEBUG "0.022666666666666595" 19:20:50 DEBUG 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG "0" 19:20:50 DEBUG "metric": { 19:20:50 DEBUG "1241.8333333333333" 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG 19:20:50 DEBUG "0.005" 19:20:50 DEBUG ] 19:20:50 DEBUG "instance": "10.142.0.107:9100" 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 DEBUG 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG }, 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG [...] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG "value": [ 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 DEBUG 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 DEBUG 19:20:50 DEBUG } 19:20:50 DEBUG 19:20:50 DEBUG } 19:20:50 DEBUG 19:20:50 DEBUG "instance": "10.142.0.43:9100" 19:20:50 DEBUG [...] 19:20:50 DEBUG }, 19:20:50 DEBUG 19:20:50 DEBUG "value": [ 19:20:50 DEBUG 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG 19:20:50 DEBUG "32744393.233333334" 19:20:50 DEBUG "instance": "10.142.0.43:9100" 19:20:50 DEBUG ] 19:20:50 DEBUG }, 19:20:50 DEBUG } 19:20:50 DEBUG "value": [ 19:20:50 DEBUG ] 19:20:50 DEBUG 1713378050, 19:20:50 DEBUG } 19:20:50 DEBUG "32749788.4" 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG ] 19:20:50 DEBUG } 19:20:50 DEBUG } 19:20:50 INFO 19:20:50 INFO [loop_until]: kubectl --namespace=xlou top pods 19:20:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:20:50 INFO [loop_until]: OK (rc = 0) 19:20:50 DEBUG --- stdout --- 19:20:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 8303m 6347Mi am-6844b8c4b7-n64np 6120m 4938Mi am-6844b8c4b7-pzj6t 5495m 4990Mi ds-cts-0 3499m 4014Mi ds-cts-1 3215m 3932Mi ds-cts-2 3677m 4134Mi ds-idrepo-0 9303m 4167Mi ds-idrepo-1 275m 4362Mi ds-idrepo-2 255m 4134Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 946Mi idm-b85f9d978-wnhxb 4m 979Mi lodemon-9886f5b6-dt5fm 2m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8043m 7875Mi 19:20:50 DEBUG --- stderr --- 19:20:50 DEBUG 19:20:50 INFO 19:20:50 INFO [loop_until]: kubectl --namespace=xlou top node 19:20:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:20:50 INFO [loop_until]: OK (rc = 0) 19:20:50 DEBUG --- stdout --- 19:20:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 81m 0% 5512Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 328m 1% 6528Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6641m 22% 7512Mi 6% gke-xlou-cdm-default-pool-1b0769c8-qkns 6015m 20% 7827Mi 7% gke-xlou-cdm-default-pool-1b0769c8-w15q 3473m 11% 6084Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 163m 0% 4929Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9511m 31% 6513Mi 5% gke-xlou-cdm-ds-0f601751-cf7b 25767m 86% 10151Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 306m 1% 5701Mi 5% gke-xlou-cdm-ds-0f601751-j488 51m 0% 1666Mi 1% gke-xlou-cdm-ds-0f601751-mc5d 3245m 10% 7073Mi 6% gke-xlou-cdm-ds-0f601751-nj6q 82m 0% 2181Mi 1% gke-xlou-cdm-ds-0f601751-nmxm 3309m 11% 5548Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 8229m 51% 11097Mi 18% 19:20:50 DEBUG --- stderr --- 19:20:50 DEBUG 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "1191.7333333333333" 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "pod": "ds-cts-0" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "8985248.135098243" 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "pod": "ds-idrepo-0" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "pod": "ds-cts-2" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "0" 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "0" 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378110" 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "pod": "ds-cts-0" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "6.2565354093260925" 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG { 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG "data": { 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "result": [ 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:21:50 DEBUG { 19:21:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:21:50 DEBUG }, 19:21:50 DEBUG "metric": { 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG "instance": "10.142.0.107:9100" 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG }, 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 DEBUG 19:21:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:21:50 DEBUG 19:21:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:21:50 DEBUG }, 19:21:50 DEBUG 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG "instance": "10.142.0.43:9100" 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG }, 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "1201.4333333333334" 19:21:50 DEBUG "value": [ 19:21:50 DEBUG "1192.8333333333333" 19:21:50 DEBUG ] 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG "35462099968" 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG ] 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "pod": "ds-idrepo-0" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "pod": "ds-idrepo-2" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "0" 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713378110" 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "pod": "ds-cts-0" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "0" 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "pod": "ds-cts-0" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "pod": "ds-cts-2" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "0" 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378110" 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 ERROR [http_cmd]: ERROR 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 ERROR --- status code --- 19:21:50 DEBUG --- status code --- 19:21:50 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 ERROR --- http response --- 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:21:50 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "pod": "ds-cts-0" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "pod": "ds-cts-2" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "0" 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG { 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "data": { 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG "result": [ 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG { 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "instance": "10.142.0.107:9100" 19:21:50 DEBUG "pod": "ds-idrepo-0" 19:21:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:21:50 DEBUG }, 19:21:50 DEBUG }, 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG "value": [ 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG [...] 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "instance": "10.142.0.43:9100" 19:21:50 DEBUG "pod": "ds-idrepo-2" 19:21:50 DEBUG "pod": "ds-cts-0" 19:21:50 DEBUG }, 19:21:50 DEBUG }, 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG "value": [ 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "0.03999999999999968" 19:21:50 DEBUG "0" 19:21:50 DEBUG "20163922.927975114" 19:21:50 DEBUG ] 19:21:50 DEBUG ] 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG ] 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713378110" 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "8374796288" 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378110" 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "instance": "10.142.0.107:9100" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "instance": "10.142.0.43:9100" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "0.03666666666666675" 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "pod": "ds-cts-0" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "pod": "ds-cts-2" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "0" 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "230922" 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378110" 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713378110" 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713378110" 19:21:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378110" 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "instance": "10.142.0.107:9100" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "instance": "10.142.0.43:9100" 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "35183127.66666667" 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG --- http response --- 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG { 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "data": { 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG "result": [ 19:21:50 DEBUG { 19:21:50 DEBUG { 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "data": { 19:21:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG }, 19:21:50 DEBUG "result": [ 19:21:50 DEBUG "value": [ 19:21:50 DEBUG { 19:21:50 DEBUG 19:21:50 DEBUG "metric": { 19:21:50 DEBUG 19:21:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:21:50 DEBUG 19:21:50 DEBUG }, 19:21:50 DEBUG [...] 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG 19:21:50 INFO [http_cmd]: http status code OK 19:21:50 DEBUG 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:21:50 DEBUG --- status code --- 19:21:50 DEBUG [...] 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG }, 19:21:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:50 DEBUG 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG "value": [ 19:21:50 DEBUG --- http response --- 19:21:50 DEBUG 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:50 DEBUG 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG "1192.8333333333333" 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG ----- output ----- 19:21:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:21:50 DEBUG { 19:21:50 DEBUG ] 19:21:50 DEBUG { 19:21:50 DEBUG { 19:21:50 DEBUG }, 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG } 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "status": "success", 19:21:50 DEBUG "value": [ 19:21:50 DEBUG "data": { 19:21:50 DEBUG ] 19:21:50 DEBUG "data": { 19:21:50 DEBUG "data": { 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG } 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "resultType": "vector", 19:21:50 DEBUG "8.03091845338342" 19:21:50 DEBUG "result": [ 19:21:50 DEBUG } 19:21:50 DEBUG "result": [ 19:21:50 DEBUG "result": [ 19:21:50 DEBUG ] 19:21:50 DEBUG { 19:21:50 DEBUG { 19:21:50 DEBUG { 19:21:50 DEBUG } 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG "metric": { 19:21:50 DEBUG ] 19:21:50 DEBUG "pod": "ds-idrepo-0" 19:21:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:21:50 DEBUG "instance": "10.142.0.107:9100" 19:21:50 DEBUG } 19:21:50 DEBUG }, 19:21:50 DEBUG }, 19:21:50 DEBUG }, 19:21:50 DEBUG } 19:21:50 DEBUG "value": [ 19:21:50 DEBUG "value": [ 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG [...] 19:21:50 DEBUG [...] 19:21:50 DEBUG [...] 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG 19:21:50 DEBUG "pod": "ds-idrepo-2" 19:21:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:21:50 DEBUG "instance": "10.142.0.43:9100" 19:21:50 DEBUG }, 19:21:50 DEBUG }, 19:21:50 DEBUG }, 19:21:50 DEBUG "value": [ 19:21:50 DEBUG "value": [ 19:21:50 DEBUG "value": [ 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG 1713378110, 19:21:50 DEBUG "0" 19:21:50 DEBUG "0.005" 19:21:50 DEBUG "35177375.266666666" 19:21:50 DEBUG ] 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG ] 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG ] 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 DEBUG } 19:21:50 INFO 19:21:50 INFO [loop_until]: kubectl --namespace=xlou top pods 19:21:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:21:50 INFO [loop_until]: OK (rc = 0) 19:21:50 DEBUG --- stdout --- 19:21:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 9689m 6471Mi am-6844b8c4b7-n64np 6965m 4995Mi am-6844b8c4b7-pzj6t 7278m 7302Mi ds-cts-0 6065m 4462Mi ds-cts-1 5908m 4442Mi ds-cts-2 6338m 4602Mi ds-idrepo-0 10039m 4210Mi ds-idrepo-1 323m 4363Mi ds-idrepo-2 243m 4135Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 946Mi idm-b85f9d978-wnhxb 4m 979Mi lodemon-9886f5b6-dt5fm 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8134m 7896Mi 19:21:50 DEBUG --- stderr --- 19:21:50 DEBUG 19:21:50 INFO 19:21:50 INFO [loop_until]: kubectl --namespace=xlou top node 19:21:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:21:50 INFO [loop_until]: OK (rc = 0) 19:21:50 DEBUG --- stdout --- 19:21:50 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 78m 0% 5505Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 362m 1% 6539Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7260m 24% 7563Mi 6% gke-xlou-cdm-default-pool-1b0769c8-qkns 6827m 22% 10139Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 6222m 20% 6577Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 4949Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10184m 34% 6551Mi 5% gke-xlou-cdm-ds-0f601751-cf7b 27172m 91% 10308Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 296m 0% 5708Mi 5% gke-xlou-cdm-ds-0f601751-j488 51m 0% 1659Mi 1% gke-xlou-cdm-ds-0f601751-mc5d 6086m 20% 7548Mi 6% gke-xlou-cdm-ds-0f601751-nj6q 75m 0% 2189Mi 1% gke-xlou-cdm-ds-0f601751-nmxm 6293m 21% 6156Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8247m 51% 11003Mi 18% 19:21:50 DEBUG --- stderr --- 19:21:50 DEBUG 19:22:45 WARNING File /tmp/lodemon_data-20240417_191547/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [17/Apr/2024 19:22:45] "GET /monitoring/average?start_time=20240417_181852&stop_time=20240417_182044 HTTP/1.1" 200 - 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG { 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "data": { 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG [...] 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "180.56666666666666" 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG { 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "data": { 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG [...] 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "0" 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG { 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "data": { 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG [...] 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "2057909.6435886933" 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG { 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "data": { 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "pod": "ds-idrepo-0" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG [...] 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "pod": "ds-cts-2" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "0" 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG { 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "data": { 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG [...] 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "2.5535582324267803" 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG --- status code --- 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378170" 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG ----- output ----- 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:22:50 DEBUG { 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG { 19:22:50 DEBUG "data": { 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG "data": { 19:22:50 DEBUG "result": [ 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG "resultType": "vector", 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG { 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG "result": [ 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG "metric": { 19:22:50 DEBUG { 19:22:50 DEBUG { 19:22:50 DEBUG { 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG { 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG "pod": "ds-idrepo-0" 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG "metric": { 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG }, 19:22:50 DEBUG "data": { 19:22:50 DEBUG "data": { 19:22:50 DEBUG "data": { 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713378170" 19:22:50 DEBUG "pod": "ds-cts-0" 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:50 DEBUG "value": [ 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG }, 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG 19:22:50 DEBUG "result": [ 19:22:50 DEBUG "result": [ 19:22:50 DEBUG "result": [ 19:22:50 DEBUG "value": [ 19:22:50 DEBUG { 19:22:50 DEBUG { 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378170" 19:22:50 DEBUG 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG { 19:22:50 DEBUG { 19:22:50 DEBUG { 19:22:50 DEBUG 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG "metric": { 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG 19:22:50 DEBUG "data": { 19:22:50 DEBUG "data": { 19:22:50 DEBUG [...] 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG "instance": "10.142.0.107:9100" 19:22:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:22:50 DEBUG 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG }, 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG }, 19:22:50 DEBUG }, 19:22:50 DEBUG [...] 19:22:50 DEBUG "result": [ 19:22:50 DEBUG "result": [ 19:22:50 DEBUG 19:22:50 DEBUG "value": [ 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG "value": [ 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG { 19:22:50 DEBUG { 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:50 DEBUG 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "pod": "ds-idrepo-2" 19:22:50 DEBUG 19:22:50 DEBUG { 19:22:50 ERROR [http_cmd]: ERROR 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG 19:22:50 DEBUG "pod": "ds-idrepo-0" 19:22:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:22:50 DEBUG }, 19:22:50 DEBUG 19:22:50 DEBUG "status": "success", 19:22:50 ERROR --- status code --- 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG { 19:22:50 DEBUG "pod": "ds-cts-2" 19:22:50 DEBUG }, 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG [...] 19:22:50 DEBUG "data": { 19:22:50 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:22:50 DEBUG [...] 19:22:50 DEBUG [...] 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG 19:22:50 DEBUG "resultType": "vector", 19:22:50 ERROR --- http response --- 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "data": { 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "0" 19:22:50 DEBUG 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG ] 19:22:50 DEBUG 19:22:50 DEBUG { 19:22:50 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG "0" 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG } 19:22:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "instance": "10.142.0.43:9100" 19:22:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:22:50 DEBUG { 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG ] 19:22:50 DEBUG [...] 19:22:50 DEBUG [...] 19:22:50 DEBUG ] 19:22:50 DEBUG }, 19:22:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:22:50 DEBUG }, 19:22:50 DEBUG }, 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "data": { 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG } 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG } 19:22:50 DEBUG "value": [ 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG "value": [ 19:22:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG ] 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG } 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG }, 19:22:50 DEBUG "result": [ 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:50 DEBUG } 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "1339.0666666666666" 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "36722772992" 19:22:50 DEBUG "value": [ 19:22:50 DEBUG { 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG } 19:22:50 DEBUG "pod": "ds-idrepo-2" 19:22:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:22:50 DEBUG ] 19:22:50 DEBUG "147.93333333333334" 19:22:50 DEBUG 19:22:50 DEBUG ] 19:22:50 DEBUG 19:22:50 DEBUG "metric": { 19:22:50 DEBUG { 19:22:50 DEBUG }, 19:22:50 DEBUG }, 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG [...] 19:22:50 DEBUG } 19:22:50 DEBUG 19:22:50 DEBUG "pod": "ds-cts-0" 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "value": [ 19:22:50 DEBUG "value": [ 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG 19:22:50 DEBUG ] 19:22:50 DEBUG 19:22:50 DEBUG }, 19:22:50 DEBUG "data": { 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG 19:22:50 DEBUG } 19:22:50 DEBUG [...] 19:22:50 DEBUG "value": [ 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "0" 19:22:50 DEBUG "0" 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 DEBUG 19:22:50 DEBUG } 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "metric": { 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG }, 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "instance": "10.142.0.107:9100" 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG "value": [ 19:22:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:22:50 DEBUG [...] 19:22:50 DEBUG }, 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG }, 19:22:50 DEBUG 19:22:50 DEBUG "value": [ 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 DEBUG "211483.5" 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG } 19:22:50 DEBUG "93431671.46666665" 19:22:50 DEBUG "pod": "ds-cts-2" 19:22:50 DEBUG 19:22:50 DEBUG ] 19:22:50 DEBUG ] 19:22:50 DEBUG }, 19:22:50 DEBUG [...] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG 19:22:50 DEBUG } 19:22:50 DEBUG "0" 19:22:50 DEBUG 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG "instance": "10.142.0.43:9100" 19:22:50 DEBUG } 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713378170" 19:22:50 DEBUG }, 19:22:50 DEBUG ] 19:22:50 DEBUG "value": [ 19:22:50 DEBUG } 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG } 19:22:50 DEBUG "0.02913333333333341" 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG { 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "data": { 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG [...] 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "8685899776" 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378170" 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG { 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "data": { 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "instance": "10.142.0.107:9100" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG [...] 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "instance": "10.142.0.43:9100" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "0.023" 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG { 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "data": { 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "pod": "ds-cts-0" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG [...] 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "pod": "ds-cts-2" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "0.00025597" 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378170" 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713378170" 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713378170" 19:22:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378170" 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG { 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "data": { 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "instance": "10.142.0.107:9100" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG [...] 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "instance": "10.142.0.43:9100" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "8415571.8" 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG { 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "data": { 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG [...] 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "181.56666666666666" 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG { 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "data": { 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG [...] 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "6.081231605115179" 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG { 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "data": { 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "pod": "ds-idrepo-0" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG [...] 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG 19:22:50 DEBUG "pod": "ds-idrepo-2" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "0" 19:22:50 DEBUG ] 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:50 DEBUG } 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG ] 19:22:50 DEBUG { 19:22:50 DEBUG } 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG } 19:22:50 DEBUG "data": { 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG [...] 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "0.006" 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 INFO [http_cmd]: http status code OK 19:22:50 DEBUG --- status code --- 19:22:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:50 DEBUG --- http response --- 19:22:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:50 DEBUG ----- output ----- 19:22:50 DEBUG { 19:22:50 DEBUG "status": "success", 19:22:50 DEBUG "data": { 19:22:50 DEBUG "resultType": "vector", 19:22:50 DEBUG "result": [ 19:22:50 DEBUG { 19:22:50 DEBUG "metric": { 19:22:50 DEBUG "instance": "10.142.0.107:9100" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG [...] 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG 19:22:50 DEBUG "instance": "10.142.0.43:9100" 19:22:50 DEBUG }, 19:22:50 DEBUG "value": [ 19:22:50 DEBUG 1713378170, 19:22:50 DEBUG "8408796.066666666" 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG ] 19:22:50 DEBUG } 19:22:50 DEBUG } 19:22:50 INFO 19:22:50 INFO [loop_until]: kubectl --namespace=xlou top pods 19:22:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:22:50 INFO [loop_until]: OK (rc = 0) 19:22:50 DEBUG --- stdout --- 19:22:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 6700m 6466Mi am-6844b8c4b7-n64np 4792m 7354Mi am-6844b8c4b7-pzj6t 3906m 7358Mi ds-cts-0 5094m 4666Mi ds-cts-1 4698m 4501Mi ds-cts-2 4940m 4683Mi ds-idrepo-0 6647m 4220Mi ds-idrepo-1 108m 4363Mi ds-idrepo-2 159m 4135Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 946Mi idm-b85f9d978-wnhxb 4m 979Mi lodemon-9886f5b6-dt5fm 6m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 6035m 8191Mi 19:22:50 DEBUG --- stderr --- 19:22:50 DEBUG 19:22:50 INFO 19:22:50 INFO [loop_until]: kubectl --namespace=xlou top node 19:22:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:22:51 INFO [loop_until]: OK (rc = 0) 19:22:51 DEBUG --- stdout --- 19:22:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 78m 0% 5508Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 144m 0% 6538Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 3576m 11% 9941Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 3274m 10% 10218Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 4864m 16% 6768Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 139m 0% 4941Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 4729m 15% 6560Mi 5% gke-xlou-cdm-ds-0f601751-cf7b 15271m 51% 10314Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 163m 0% 5709Mi 5% gke-xlou-cdm-ds-0f601751-j488 51m 0% 1671Mi 1% gke-xlou-cdm-ds-0f601751-mc5d 4847m 16% 7622Mi 6% gke-xlou-cdm-ds-0f601751-nj6q 77m 0% 2172Mi 1% gke-xlou-cdm-ds-0f601751-nmxm 5096m 17% 6249Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 4278m 26% 3829Mi 6% 19:22:51 DEBUG --- stderr --- 19:22:51 DEBUG 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "824.5" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "0" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "1.595018431095216" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "pod": "ds-idrepo-0" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:50 DEBUG 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG 19:23:50 DEBUG { 19:23:50 DEBUG "pod": "ds-cts-2" 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG }, 19:23:50 DEBUG "data": { 19:23:50 DEBUG "value": [ 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "result": [ 19:23:50 DEBUG "0" 19:23:50 DEBUG { 19:23:50 DEBUG ] 19:23:50 DEBUG "metric": { 19:23:50 DEBUG } 19:23:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:23:50 DEBUG ] 19:23:50 DEBUG }, 19:23:50 DEBUG } 19:23:50 DEBUG "value": [ 19:23:50 DEBUG } 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "291699.8064113475" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "pod": "ds-idrepo-0" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "pod": "ds-idrepo-2" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "0" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713378230" 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 ERROR [http_cmd]: ERROR 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 ERROR --- status code --- 19:23:50 DEBUG --- status code --- 19:23:50 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 ERROR --- http response --- 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:50 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "1385.1666666666667" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "857.5333333333333" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378230" 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378230" 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "pod": "ds-idrepo-0" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "pod": "ds-idrepo-2" 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:23:50 DEBUG }, 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG "value": [ 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG { 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "0" 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG "data": { 19:23:50 DEBUG ] 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713378230" 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378230" 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG } 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG "result": [ 19:23:50 DEBUG ] 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG { 19:23:50 DEBUG { 19:23:50 DEBUG { 19:23:50 DEBUG } 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "metric": { 19:23:50 DEBUG } 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:50 DEBUG "data": { 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:50 DEBUG "data": { 19:23:50 DEBUG "pod": "ds-cts-0" 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG }, 19:23:50 DEBUG { 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG { 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "result": [ 19:23:50 DEBUG "value": [ 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG { 19:23:50 DEBUG 19:23:50 DEBUG "data": { 19:23:50 DEBUG "data": { 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "data": { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG "instance": "10.142.0.107:9100" 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:23:50 DEBUG 19:23:50 DEBUG "result": [ 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG }, 19:23:50 DEBUG "result": [ 19:23:50 DEBUG }, 19:23:50 DEBUG [...] 19:23:50 DEBUG { 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "value": [ 19:23:50 DEBUG { 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "data": { 19:23:50 DEBUG 19:23:50 DEBUG "metric": { 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "pod": "ds-cts-0" 19:23:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG 19:23:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG }, 19:23:50 DEBUG }, 19:23:50 DEBUG "result": [ 19:23:50 DEBUG 19:23:50 DEBUG }, 19:23:50 DEBUG 19:23:50 DEBUG "pod": "ds-cts-2" 19:23:50 DEBUG "value": [ 19:23:50 DEBUG "value": [ 19:23:50 DEBUG { 19:23:50 DEBUG [...] 19:23:50 DEBUG "value": [ 19:23:50 DEBUG [...] 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG }, 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "metric": { 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "instance": "10.142.0.107:9100" 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG }, 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:50 DEBUG 19:23:50 DEBUG "value": [ 19:23:50 DEBUG "instance": "10.142.0.43:9100" 19:23:50 DEBUG [...] 19:23:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:23:50 DEBUG "0" 19:23:50 DEBUG [...] 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG }, 19:23:50 DEBUG 19:23:50 DEBUG }, 19:23:50 DEBUG ] 19:23:50 DEBUG 19:23:50 DEBUG { 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG "value": [ 19:23:50 DEBUG } 19:23:50 DEBUG 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "pod": "ds-cts-2" 19:23:50 DEBUG 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG ] 19:23:50 DEBUG 19:23:50 DEBUG "data": { 19:23:50 DEBUG }, 19:23:50 DEBUG [...] 19:23:50 DEBUG "37595710464" 19:23:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:23:50 DEBUG "107094.65509219859" 19:23:50 DEBUG } 19:23:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG ] 19:23:50 DEBUG }, 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG }, 19:23:50 DEBUG "result": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG 19:23:50 DEBUG } 19:23:50 DEBUG "value": [ 19:23:50 DEBUG } 19:23:50 DEBUG "value": [ 19:23:50 DEBUG { 19:23:50 DEBUG "0" 19:23:50 DEBUG 19:23:50 DEBUG ] 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG ] 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "metric": { 19:23:50 DEBUG ] 19:23:50 DEBUG "instance": "10.142.0.43:9100" 19:23:50 DEBUG } 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG "3106.336680851064" 19:23:50 DEBUG } 19:23:50 DEBUG "145837.5" 19:23:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:23:50 DEBUG } 19:23:50 DEBUG }, 19:23:50 DEBUG } 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG }, 19:23:50 DEBUG ] 19:23:50 DEBUG "value": [ 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 DEBUG "value": [ 19:23:50 DEBUG } 19:23:50 DEBUG 1713378230, 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 DEBUG ] 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:50 DEBUG ] 19:23:50 DEBUG 19:23:50 DEBUG } 19:23:50 DEBUG "0.05353333333333315" 19:23:50 DEBUG } 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG } 19:23:50 DEBUG 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG { 19:23:50 DEBUG } 19:23:50 DEBUG 19:23:50 DEBUG } 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG [...] 19:23:50 DEBUG ] 19:23:50 DEBUG "data": { 19:23:50 DEBUG 19:23:50 DEBUG } 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG 19:23:50 DEBUG } 19:23:50 DEBUG "result": [ 19:23:50 DEBUG 19:23:50 DEBUG { 19:23:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:23:50 DEBUG "metric": { 19:23:50 DEBUG }, 19:23:50 DEBUG "instance": "10.142.0.107:9100" 19:23:50 DEBUG "value": [ 19:23:50 DEBUG }, 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG "574717952" 19:23:50 DEBUG 19:23:50 DEBUG ] 19:23:50 DEBUG 19:23:50 DEBUG } 19:23:50 DEBUG 19:23:50 DEBUG ] 19:23:50 DEBUG [...] 19:23:50 DEBUG } 19:23:50 DEBUG 19:23:50 DEBUG } 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "instance": "10.142.0.43:9100" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "0.04199999999999997" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "pod": "ds-cts-0" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "pod": "ds-cts-2" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "0" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378230" 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "0.006" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "instance": "10.142.0.107:9100" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "instance": "10.142.0.43:9100" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "30946032.733333334" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713378230" 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "824.0666666666666" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378230" 19:23:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378230" 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "2.7172242351239064" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "pod": "ds-idrepo-0" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "pod": "ds-idrepo-2" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "0" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 INFO [http_cmd]: http status code OK 19:23:50 DEBUG --- status code --- 19:23:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:50 DEBUG --- http response --- 19:23:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:50 DEBUG ----- output ----- 19:23:50 DEBUG { 19:23:50 DEBUG "status": "success", 19:23:50 DEBUG "data": { 19:23:50 DEBUG "resultType": "vector", 19:23:50 DEBUG "result": [ 19:23:50 DEBUG { 19:23:50 DEBUG "metric": { 19:23:50 DEBUG "instance": "10.142.0.107:9100" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG [...] 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG 19:23:50 DEBUG "instance": "10.142.0.43:9100" 19:23:50 DEBUG }, 19:23:50 DEBUG "value": [ 19:23:50 DEBUG 1713378230, 19:23:50 DEBUG "30940770.6" 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG ] 19:23:50 DEBUG } 19:23:50 DEBUG } 19:23:50 INFO 19:23:50 INFO [loop_until]: kubectl --namespace=xlou top pods 19:23:50 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:23:50 INFO [loop_until]: OK (rc = 0) 19:23:50 DEBUG --- stdout --- 19:23:50 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 2939m 6405Mi am-6844b8c4b7-n64np 2102m 7366Mi am-6844b8c4b7-pzj6t 2342m 7361Mi ds-cts-0 3433m 4681Mi ds-cts-1 3426m 4550Mi ds-cts-2 4271m 4707Mi ds-idrepo-0 3393m 4221Mi ds-idrepo-1 50m 4364Mi ds-idrepo-2 87m 4136Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 946Mi idm-b85f9d978-wnhxb 4m 979Mi lodemon-9886f5b6-dt5fm 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 3204m 456Mi 19:23:50 DEBUG --- stderr --- 19:23:50 DEBUG 19:23:51 INFO 19:23:51 INFO [loop_until]: kubectl --namespace=xlou top node 19:23:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:23:51 INFO [loop_until]: OK (rc = 0) 19:23:51 DEBUG --- stdout --- 19:23:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 5494Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 178m 0% 6546Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 3504m 11% 9949Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 3400m 11% 10207Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 4025m 13% 6802Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 163m 0% 4944Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 4465m 14% 6573Mi 5% gke-xlou-cdm-ds-0f601751-cf7b 5938m 19% 9331Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 154m 0% 5722Mi 5% gke-xlou-cdm-ds-0f601751-j488 52m 0% 1673Mi 1% gke-xlou-cdm-ds-0f601751-mc5d 3687m 12% 7646Mi 6% gke-xlou-cdm-ds-0f601751-nj6q 83m 0% 2176Mi 1% gke-xlou-cdm-ds-0f601751-nmxm 3591m 12% 6270Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 3469m 21% 4348Mi 7% 19:23:51 DEBUG --- stderr --- 19:23:51 DEBUG 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "1226.7" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "0" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "pod": "ds-idrepo-0" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "pod": "ds-idrepo-2" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "0" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "4.244224707957061" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "pod": "ds-idrepo-0" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "pod": "ds-cts-2" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "0" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713378290" 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "203870605.7256697" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 ERROR [http_cmd]: ERROR 19:24:50 ERROR --- status code --- 19:24:50 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:24:50 ERROR --- http response --- 19:24:50 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:24:50 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "1299.9" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378290" 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "1226.0666666666666" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "instance": "10.142.0.107:9100" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "instance": "10.142.0.43:9100" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "0.03893333333333366" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378290" 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "139956" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "pod": "ds-cts-0" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "pod": "ds-cts-2" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "0.000017330000000000002" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "instance": "10.142.0.107:9100" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "instance": "10.142.0.43:9100" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "0.028333333333333332" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713378290" 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378290" 19:24:50 DEBUG --- status code --- 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:24:50 DEBUG { 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG { 19:24:50 DEBUG "data": { 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG "data": { 19:24:50 DEBUG "result": [ 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG { 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG "result": [ 19:24:50 DEBUG "metric": { 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG { 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG { 19:24:50 DEBUG "pod": "ds-idrepo-0" 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG "metric": { 19:24:50 DEBUG }, 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG "data": { 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:50 DEBUG "pod": "ds-cts-0" 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:24:50 DEBUG "value": [ 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG }, 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG "value": [ 19:24:50 DEBUG { 19:24:50 DEBUG 19:24:50 DEBUG { 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "data": { 19:24:50 DEBUG { 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG 19:24:50 DEBUG "data": { 19:24:50 DEBUG [...] 19:24:50 DEBUG "data": { 19:24:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG }, 19:24:50 DEBUG "result": [ 19:24:50 DEBUG "data": { 19:24:50 DEBUG [...] 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:24:50 DEBUG "result": [ 19:24:50 DEBUG 19:24:50 DEBUG "result": [ 19:24:50 DEBUG "value": [ 19:24:50 DEBUG { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG 19:24:50 DEBUG { 19:24:50 DEBUG 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "result": [ 19:24:50 DEBUG 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "pod": "ds-idrepo-2" 19:24:50 DEBUG "metric": { 19:24:50 DEBUG 19:24:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:24:50 DEBUG { 19:24:50 DEBUG 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:24:50 DEBUG }, 19:24:50 DEBUG "instance": "10.142.0.107:9100" 19:24:50 DEBUG 19:24:50 DEBUG }, 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "pod": "ds-cts-2" 19:24:50 DEBUG "data": { 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG }, 19:24:50 DEBUG [...] 19:24:50 DEBUG "value": [ 19:24:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:24:50 DEBUG }, 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG "result": [ 19:24:50 DEBUG 19:24:50 DEBUG "0" 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG { 19:24:50 DEBUG 19:24:50 DEBUG ] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "0" 19:24:50 DEBUG "metric": { 19:24:50 DEBUG 19:24:50 DEBUG } 19:24:50 DEBUG 19:24:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG ] 19:24:50 DEBUG "pod": "ds-cts-0" 19:24:50 DEBUG [...] 19:24:50 DEBUG ] 19:24:50 DEBUG [...] 19:24:50 DEBUG }, 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG } 19:24:50 DEBUG }, 19:24:50 DEBUG 19:24:50 DEBUG } 19:24:50 DEBUG 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG ] 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG } 19:24:50 DEBUG 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG } 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "4215193600" 19:24:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:24:50 DEBUG 19:24:50 DEBUG } 19:24:50 DEBUG 19:24:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:24:50 DEBUG "instance": "10.142.0.43:9100" 19:24:50 DEBUG ] 19:24:50 DEBUG }, 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG }, 19:24:50 DEBUG }, 19:24:50 DEBUG } 19:24:50 DEBUG "value": [ 19:24:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:24:50 DEBUG [...] 19:24:50 DEBUG "value": [ 19:24:50 DEBUG "value": [ 19:24:50 DEBUG ] 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG }, 19:24:50 DEBUG 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG } 19:24:50 DEBUG "1226.4666666666667" 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG "2766724.0493355133" 19:24:50 DEBUG "39511729152" 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG 19:24:50 DEBUG ] 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG "0" 19:24:50 DEBUG "pod": "ds-cts-2" 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG ] 19:24:50 DEBUG }, 19:24:50 DEBUG ] 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 DEBUG "value": [ 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 DEBUG "0" 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378290" 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:24:50 DEBUG }, 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG "value": [ 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG "0.006" 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "instance": "10.142.0.107:9100" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "instance": "10.142.0.43:9100" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "34429841.06666666" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713378290" 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378290" 19:24:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378290" 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "8.03393694721978" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "pod": "ds-idrepo-0" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "pod": "ds-idrepo-2" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "0" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:50 INFO [http_cmd]: http status code OK 19:24:50 DEBUG --- status code --- 19:24:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:50 DEBUG --- http response --- 19:24:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:50 DEBUG ----- output ----- 19:24:50 DEBUG { 19:24:50 DEBUG "status": "success", 19:24:50 DEBUG "data": { 19:24:50 DEBUG "resultType": "vector", 19:24:50 DEBUG "result": [ 19:24:50 DEBUG { 19:24:50 DEBUG "metric": { 19:24:50 DEBUG "instance": "10.142.0.107:9100" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG [...] 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG 19:24:50 DEBUG "instance": "10.142.0.43:9100" 19:24:50 DEBUG }, 19:24:50 DEBUG "value": [ 19:24:50 DEBUG 1713378290, 19:24:50 DEBUG "34424797.766666666" 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG ] 19:24:50 DEBUG } 19:24:50 DEBUG } 19:24:51 INFO 19:24:51 INFO [loop_until]: kubectl --namespace=xlou top pods 19:24:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:24:51 INFO [loop_until]: OK (rc = 0) 19:24:51 DEBUG --- stdout --- 19:24:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 9661m 6459Mi am-6844b8c4b7-n64np 6521m 7368Mi am-6844b8c4b7-pzj6t 5872m 7384Mi ds-cts-0 5972m 4675Mi ds-cts-1 5671m 4540Mi ds-cts-2 5864m 4690Mi ds-idrepo-0 9781m 4213Mi ds-idrepo-1 253m 4365Mi ds-idrepo-2 300m 4137Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 10m 948Mi idm-b85f9d978-wnhxb 4m 979Mi lodemon-9886f5b6-dt5fm 9m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8009m 3930Mi 19:24:51 DEBUG --- stderr --- 19:24:51 DEBUG 19:24:51 INFO 19:24:51 INFO [loop_until]: kubectl --namespace=xlou top node 19:24:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:24:51 INFO [loop_until]: OK (rc = 0) 19:24:51 DEBUG --- stdout --- 19:24:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 76m 0% 5489Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 297m 0% 6546Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6742m 22% 9933Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 6267m 20% 10221Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 5822m 19% 6799Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 150m 0% 4927Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 10217m 34% 6563Mi 5% gke-xlou-cdm-ds-0f601751-cf7b 25840m 86% 10311Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 357m 1% 5716Mi 5% gke-xlou-cdm-ds-0f601751-j488 50m 0% 1675Mi 1% gke-xlou-cdm-ds-0f601751-mc5d 5555m 18% 7664Mi 6% gke-xlou-cdm-ds-0f601751-nj6q 77m 0% 2173Mi 1% gke-xlou-cdm-ds-0f601751-nmxm 5835m 19% 6270Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8190m 51% 7523Mi 12% 19:24:51 DEBUG --- stderr --- 19:24:51 DEBUG 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "1304.9666666666667" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "0" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG { 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "pod": "ds-idrepo-0" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "pod": "ds-idrepo-2" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "0" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "6.237832916338201" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "pod": "ds-idrepo-0" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "pod": "ds-cts-2" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "0" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713378350" 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 DEBUG --- status code --- 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:50 ERROR [http_cmd]: ERROR 19:25:50 DEBUG { 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 ERROR --- status code --- 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:25:50 DEBUG "data": { 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 ERROR --- http response --- 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG { 19:25:50 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG }, 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:25:50 DEBUG "value": [ 19:25:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:25:50 DEBUG }, 19:25:50 DEBUG 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG }, 19:25:50 DEBUG 19:25:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:25:50 DEBUG "value": [ 19:25:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:25:50 DEBUG }, 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG "110.46666666666667" 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG ] 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "156388" 19:25:50 DEBUG } 19:25:50 DEBUG "72859.58117695445" 19:25:50 DEBUG ] 19:25:50 DEBUG ] 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "1276.0666666666666" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378350" 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "instance": "10.142.0.107:9100" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "instance": "10.142.0.43:9100" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "0.028133333333334083" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG { 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "0.006" 19:25:50 DEBUG ] 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713378350" 19:25:50 DEBUG } 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:25:50 DEBUG ] 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG } 19:25:50 DEBUG { 19:25:50 DEBUG } 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "4311911.43709086" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "pod": "ds-cts-0" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "pod": "ds-cts-2" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "0" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "instance": "10.142.0.107:9100" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "instance": "10.142.0.43:9100" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "0.021000000000000053" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "7061323776" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:50 DEBUG { 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "data": { 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "result": [ 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "pod": "ds-idrepo-0" 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG }, 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:25:50 DEBUG "value": [ 19:25:50 DEBUG "pod": "ds-cts-0" 19:25:50 DEBUG }, 19:25:50 DEBUG 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "pod": "ds-idrepo-2" 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG }, 19:25:50 DEBUG 19:25:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:25:50 DEBUG "value": [ 19:25:50 DEBUG "pod": "ds-cts-2" 19:25:50 DEBUG }, 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG "0" 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG ] 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "1304.0333333333333" 19:25:50 DEBUG } 19:25:50 DEBUG "0" 19:25:50 DEBUG ] 19:25:50 DEBUG ] 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378350" 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "instance": "10.142.0.107:9100" 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "pod": "ds-cts-0" 19:25:50 DEBUG }, 19:25:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "instance": "10.142.0.43:9100" 19:25:50 DEBUG 19:25:50 DEBUG "pod": "ds-cts-2" 19:25:50 DEBUG }, 19:25:50 DEBUG "pod": "am-6844b8c4b7-n64np" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "40858227712" 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "0" 19:25:50 DEBUG ] 19:25:50 DEBUG "0" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "instance": "10.142.0.107:9100" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "instance": "10.142.0.43:9100" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "29675868.2" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378350" 19:25:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378350" 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "8.034965837780145" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "instance": "10.142.0.107:9100" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "instance": "10.142.0.43:9100" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "29670458.366666667" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:50 INFO [http_cmd]: http status code OK 19:25:50 DEBUG --- status code --- 19:25:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:50 DEBUG --- http response --- 19:25:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:50 DEBUG ----- output ----- 19:25:50 DEBUG { 19:25:50 DEBUG "status": "success", 19:25:50 DEBUG "data": { 19:25:50 DEBUG "resultType": "vector", 19:25:50 DEBUG "result": [ 19:25:50 DEBUG { 19:25:50 DEBUG "metric": { 19:25:50 DEBUG "pod": "ds-idrepo-0" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG [...] 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG 19:25:50 DEBUG "pod": "ds-idrepo-2" 19:25:50 DEBUG }, 19:25:50 DEBUG "value": [ 19:25:50 DEBUG 1713378350, 19:25:50 DEBUG "0" 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG ] 19:25:50 DEBUG } 19:25:50 DEBUG } 19:25:51 INFO 19:25:51 INFO [loop_until]: kubectl --namespace=xlou top pods 19:25:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:25:51 INFO [loop_until]: OK (rc = 0) 19:25:51 DEBUG --- stdout --- 19:25:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 9529m 8347Mi am-6844b8c4b7-n64np 5730m 7362Mi am-6844b8c4b7-pzj6t 5333m 7384Mi ds-cts-0 4138m 4696Mi ds-cts-1 4095m 4557Mi ds-cts-2 4277m 4714Mi ds-idrepo-0 9820m 4217Mi ds-idrepo-1 241m 4365Mi ds-idrepo-2 260m 4137Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 948Mi idm-b85f9d978-wnhxb 4m 979Mi lodemon-9886f5b6-dt5fm 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7938m 6644Mi 19:25:51 DEBUG --- stderr --- 19:25:51 DEBUG 19:25:51 INFO 19:25:51 INFO [loop_until]: kubectl --namespace=xlou top node 19:25:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:25:51 INFO [loop_until]: OK (rc = 0) 19:25:51 DEBUG --- stdout --- 19:25:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 80m 0% 5499Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 304m 1% 6548Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6321m 21% 9930Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 5468m 18% 10216Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 4386m 14% 6800Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 146m 0% 4937Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9881m 33% 6562Mi 5% gke-xlou-cdm-ds-0f601751-cf7b 26927m 90% 12211Mi 10% gke-xlou-cdm-ds-0f601751-hmm7 320m 1% 5711Mi 5% gke-xlou-cdm-ds-0f601751-j488 49m 0% 1660Mi 1% gke-xlou-cdm-ds-0f601751-mc5d 4492m 15% 7662Mi 6% gke-xlou-cdm-ds-0f601751-nj6q 79m 0% 2173Mi 1% gke-xlou-cdm-ds-0f601751-nmxm 4609m 15% 6291Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8267m 52% 10012Mi 17% 19:25:51 DEBUG --- stderr --- 19:25:51 DEBUG 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "0" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "1286.7" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "pod": "ds-idrepo-0" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "ds-cts-2" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "0" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "pod": "ds-idrepo-0" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "ds-idrepo-2" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "0" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG { 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:50 DEBUG "result": [ 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG }, 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "value": [ 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "instance": "10.142.0.107:9100" 19:26:50 DEBUG [...] 19:26:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:26:50 DEBUG }, 19:26:50 DEBUG 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG }, 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG "value": [ 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "750.6333333333333" 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG ] 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG 19:26:50 DEBUG "instance": "10.142.0.43:9100" 19:26:50 DEBUG } 19:26:50 DEBUG { 19:26:50 DEBUG "pod": "ds-idrepo-2" 19:26:50 DEBUG }, 19:26:50 DEBUG ] 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG } 19:26:50 DEBUG "data": { 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG } 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "0.030633333333332638" 19:26:50 DEBUG "result": [ 19:26:50 DEBUG "0" 19:26:50 DEBUG ] 19:26:50 DEBUG { 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG "metric": { 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG }, 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 DEBUG "value": [ 19:26:50 DEBUG } 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "1307.1333333333334" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "ds-idrepo-2" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "0" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 ERROR [http_cmd]: ERROR 19:26:50 ERROR --- status code --- 19:26:50 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:26:50 ERROR --- http response --- 19:26:50 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:26:50 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "219650.5" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "pod": "ds-cts-0" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "ds-cts-2" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "0.000017330000000000002" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713378410" 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "0.006" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:26:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:26:50 DEBUG }, 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:26:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:26:50 DEBUG }, 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "0" 19:26:50 DEBUG "0" 19:26:50 DEBUG ] 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "instance": "10.142.0.107:9100" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "instance": "10.142.0.43:9100" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "0.02300000000000004" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "pod": "ds-cts-0" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "ds-cts-2" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "0" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "8252502016" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713378410" 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "instance": "10.142.0.107:9100" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "instance": "10.142.0.43:9100" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "42071704576" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "8.026877312031937" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "pod": "ds-idrepo-0" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "ds-idrepo-2" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "0" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "pod": "ds-cts-0" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "ds-cts-2" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "0.06666666666666667" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378410" 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "1286.2666666666667" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "instance": "10.142.0.107:9100" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "instance": "10.142.0.43:9100" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "34141617.93333333" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378410" 19:26:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378410" 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "pod": "ds-idrepo-0" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "pod": "ds-idrepo-2" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "0" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:50 INFO [http_cmd]: http status code OK 19:26:50 DEBUG --- status code --- 19:26:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:50 DEBUG --- http response --- 19:26:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:50 DEBUG ----- output ----- 19:26:50 DEBUG { 19:26:50 DEBUG "status": "success", 19:26:50 DEBUG "data": { 19:26:50 DEBUG "resultType": "vector", 19:26:50 DEBUG "result": [ 19:26:50 DEBUG { 19:26:50 DEBUG "metric": { 19:26:50 DEBUG "instance": "10.142.0.107:9100" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG [...] 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG 19:26:50 DEBUG "instance": "10.142.0.43:9100" 19:26:50 DEBUG }, 19:26:50 DEBUG "value": [ 19:26:50 DEBUG 1713378410, 19:26:50 DEBUG "34136130" 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG ] 19:26:50 DEBUG } 19:26:50 DEBUG } 19:26:51 INFO 19:26:51 INFO [loop_until]: kubectl --namespace=xlou top pods 19:26:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:26:51 INFO [loop_until]: OK (rc = 0) 19:26:51 DEBUG --- stdout --- 19:26:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 7965m 8296Mi am-6844b8c4b7-n64np 5674m 7352Mi am-6844b8c4b7-pzj6t 5256m 7371Mi ds-cts-0 3546m 4727Mi ds-cts-1 3448m 4588Mi ds-cts-2 3111m 4743Mi ds-idrepo-0 9261m 4219Mi ds-idrepo-1 244m 4367Mi ds-idrepo-2 230m 4138Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 7m 948Mi idm-b85f9d978-wnhxb 4m 979Mi lodemon-9886f5b6-dt5fm 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7920m 7779Mi 19:26:51 DEBUG --- stderr --- 19:26:51 DEBUG 19:26:51 INFO 19:26:51 INFO [loop_until]: kubectl --namespace=xlou top node 19:26:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:26:51 INFO [loop_until]: OK (rc = 0) 19:26:51 DEBUG --- stdout --- 19:26:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 77m 0% 5500Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 318m 1% 6539Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6464m 21% 9928Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 5781m 19% 10214Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 3515m 11% 6877Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 4944Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 9500m 31% 6562Mi 5% gke-xlou-cdm-ds-0f601751-cf7b 24886m 83% 12078Mi 10% gke-xlou-cdm-ds-0f601751-hmm7 288m 0% 5714Mi 5% gke-xlou-cdm-ds-0f601751-j488 49m 0% 1665Mi 1% gke-xlou-cdm-ds-0f601751-mc5d 3054m 10% 7727Mi 6% gke-xlou-cdm-ds-0f601751-nj6q 80m 0% 2180Mi 1% gke-xlou-cdm-ds-0f601751-nmxm 3139m 10% 6344Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8279m 52% 10647Mi 18% 19:26:51 DEBUG --- stderr --- 19:26:51 DEBUG 19:27:33 WARNING File /tmp/lodemon_data-20240417_191547/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [17/Apr/2024 19:27:33] "GET /monitoring/average?start_time=20240417_182445&stop_time=20240417_182532 HTTP/1.1" 200 - 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG { 19:27:50 DEBUG { 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "data": { 19:27:50 DEBUG "data": { 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "result": [ 19:27:50 DEBUG "result": [ 19:27:50 DEBUG { 19:27:50 DEBUG { 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:27:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:27:50 DEBUG }, 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG [...] 19:27:50 DEBUG [...] 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:27:50 DEBUG "pod": "ds-cts-0" 19:27:50 DEBUG }, 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "405.06666666666666" 19:27:50 DEBUG "588282.1057998725" 19:27:50 DEBUG ] 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 DEBUG ] 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG { 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "data": { 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "result": [ 19:27:50 DEBUG { 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG [...] 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "0" 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG { 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "data": { 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "result": [ 19:27:50 DEBUG { 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "pod": "ds-idrepo-0" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG [...] 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG "pod": "ds-cts-2" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "0" 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG { 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "data": { 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "result": [ 19:27:50 DEBUG { 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG [...] 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "215570" 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378470" 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG { 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "data": { 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "result": [ 19:27:50 DEBUG { 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "pod": "ds-idrepo-0" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG [...] 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG "pod": "ds-idrepo-2" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "0" 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG { 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "data": { 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "result": [ 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG --- status code --- 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG { 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG "metric": { 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG "pod": "ds-cts-0" 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:50 DEBUG }, 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:27:50 DEBUG "value": [ 19:27:50 DEBUG { 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG { 19:27:50 DEBUG 19:27:50 DEBUG "data": { 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "data": { 19:27:50 DEBUG [...] 19:27:50 DEBUG "result": [ 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG 19:27:50 DEBUG { 19:27:50 DEBUG "result": [ 19:27:50 DEBUG 19:27:50 DEBUG "metric": { 19:27:50 DEBUG { 19:27:50 DEBUG 19:27:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "pod": "ds-cts-2" 19:27:50 DEBUG }, 19:27:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG "0" 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG ] 19:27:50 DEBUG [...] 19:27:50 DEBUG 19:27:50 DEBUG } 19:27:50 DEBUG 19:27:50 DEBUG [...] 19:27:50 DEBUG ] 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG } 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG } 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:27:50 DEBUG 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG }, 19:27:50 DEBUG "pod": "ds-cts-0" 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG "value": [ 19:27:50 DEBUG }, 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG "1216.9" 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG { 19:27:50 DEBUG ] 19:27:50 DEBUG "3.2178473804124916" 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG } 19:27:50 DEBUG ] 19:27:50 DEBUG "data": { 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG } 19:27:50 DEBUG ] 19:27:50 DEBUG "result": [ 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 DEBUG { 19:27:50 DEBUG } 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713378470" 19:27:50 DEBUG "metric": { 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG }, 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG "value": [ 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG [...] 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "371.93333333333334" 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG { 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "data": { 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "result": [ 19:27:50 DEBUG { 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "instance": "10.142.0.107:9100" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG [...] 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG "instance": "10.142.0.43:9100" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "0.01523333333333312" 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 ERROR [http_cmd]: ERROR 19:27:50 ERROR --- status code --- 19:27:50 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:27:50 ERROR --- http response --- 19:27:50 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:27:50 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378470" 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG --- http response --- 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG { 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "data": { 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "result": [ 19:27:50 DEBUG { 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG [...] 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "0.006" 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG { 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "data": { 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "result": [ 19:27:50 DEBUG { 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "instance": "10.142.0.107:9100" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG [...] 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG "instance": "10.142.0.43:9100" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "43376169984" 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713378470" 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713378470" 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378470" 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG { 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "data": { 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "result": [ 19:27:50 DEBUG { 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG [...] 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "112246784" 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG { 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "data": { 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "result": [ 19:27:50 DEBUG { 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG [...] 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:50 DEBUG 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG "pod": "ds-cts-0" 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG { 19:27:50 DEBUG }, 19:27:50 DEBUG { 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "value": [ 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "data": { 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "data": { 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "0" 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "result": [ 19:27:50 DEBUG ] 19:27:50 DEBUG "result": [ 19:27:50 DEBUG { 19:27:50 DEBUG } 19:27:50 DEBUG { 19:27:50 DEBUG "metric": { 19:27:50 DEBUG ] 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:27:50 DEBUG } 19:27:50 DEBUG "pod": "ds-cts-0" 19:27:50 DEBUG }, 19:27:50 DEBUG } 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG [...] 19:27:50 DEBUG [...] 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:27:50 DEBUG "pod": "ds-cts-2" 19:27:50 DEBUG }, 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "4.132823418187402" 19:27:50 DEBUG "0" 19:27:50 DEBUG ] 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 DEBUG ] 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG { 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "data": { 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "result": [ 19:27:50 DEBUG { 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG [...] 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG "pod": "ds-cts-0" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "17176328.708731677" 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 INFO [http_cmd]: http status code OK 19:27:50 DEBUG --- status code --- 19:27:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:50 DEBUG --- http response --- 19:27:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:50 DEBUG ----- output ----- 19:27:50 DEBUG { 19:27:50 DEBUG "status": "success", 19:27:50 DEBUG "data": { 19:27:50 DEBUG "resultType": "vector", 19:27:50 DEBUG "result": [ 19:27:50 DEBUG { 19:27:50 DEBUG "metric": { 19:27:50 DEBUG "instance": "10.142.0.107:9100" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG [...] 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG 19:27:50 DEBUG "instance": "10.142.0.43:9100" 19:27:50 DEBUG }, 19:27:50 DEBUG "value": [ 19:27:50 DEBUG 1713378470, 19:27:50 DEBUG "0.0113333333333333" 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG ] 19:27:50 DEBUG } 19:27:50 DEBUG } 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378470" 19:27:51 INFO [http_cmd]: http status code OK 19:27:51 INFO [http_cmd]: http status code OK 19:27:51 INFO [http_cmd]: http status code OK 19:27:51 DEBUG --- status code --- 19:27:51 DEBUG --- status code --- 19:27:51 DEBUG --- status code --- 19:27:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378471" 19:27:51 DEBUG --- http response --- 19:27:51 DEBUG --- http response --- 19:27:51 DEBUG --- http response --- 19:27:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:27:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:51 DEBUG ----- output ----- 19:27:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:51 DEBUG ----- output ----- 19:27:51 DEBUG { 19:27:51 DEBUG ----- output ----- 19:27:51 DEBUG { 19:27:51 DEBUG "status": "success", 19:27:51 DEBUG { 19:27:51 DEBUG "status": "success", 19:27:51 DEBUG "data": { 19:27:51 DEBUG "status": "success", 19:27:51 DEBUG "data": { 19:27:51 DEBUG "resultType": "vector", 19:27:51 DEBUG "data": { 19:27:51 DEBUG "resultType": "vector", 19:27:51 DEBUG "result": [ 19:27:51 DEBUG "resultType": "vector", 19:27:51 DEBUG "result": [ 19:27:51 DEBUG { 19:27:51 DEBUG "result": [ 19:27:51 DEBUG { 19:27:51 DEBUG "metric": { 19:27:51 DEBUG { 19:27:51 DEBUG "metric": { 19:27:51 DEBUG "pod": "ds-cts-0" 19:27:51 DEBUG "metric": { 19:27:51 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:27:51 DEBUG }, 19:27:51 DEBUG "pod": "ds-idrepo-0" 19:27:51 DEBUG }, 19:27:51 DEBUG "value": [ 19:27:51 DEBUG }, 19:27:51 DEBUG "value": [ 19:27:51 DEBUG 19:27:51 DEBUG "value": [ 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG [...] 19:27:51 DEBUG 19:27:51 DEBUG [...] 19:27:51 DEBUG 19:27:51 DEBUG [...] 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG "pod": "ds-cts-2" 19:27:51 DEBUG 19:27:51 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:27:51 DEBUG }, 19:27:51 DEBUG "pod": "ds-idrepo-2" 19:27:51 DEBUG }, 19:27:51 DEBUG "value": [ 19:27:51 DEBUG }, 19:27:51 DEBUG "value": [ 19:27:51 DEBUG 1713378470, 19:27:51 DEBUG "value": [ 19:27:51 DEBUG 1713378470, 19:27:51 DEBUG "0" 19:27:51 DEBUG 1713378470, 19:27:51 DEBUG "405.3666666666667" 19:27:51 DEBUG ] 19:27:51 DEBUG "0" 19:27:51 DEBUG ] 19:27:51 DEBUG } 19:27:51 DEBUG ] 19:27:51 DEBUG } 19:27:51 DEBUG ] 19:27:51 DEBUG } 19:27:51 DEBUG ] 19:27:51 DEBUG } 19:27:51 DEBUG ] 19:27:51 DEBUG } 19:27:51 DEBUG } 19:27:51 DEBUG } 19:27:51 DEBUG } 19:27:51 DEBUG } 19:27:51 INFO [http_cmd]: http status code OK 19:27:51 DEBUG --- status code --- 19:27:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:51 DEBUG --- http response --- 19:27:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:51 DEBUG ----- output ----- 19:27:51 DEBUG { 19:27:51 DEBUG "status": "success", 19:27:51 DEBUG "data": { 19:27:51 DEBUG "resultType": "vector", 19:27:51 DEBUG "result": [ 19:27:51 DEBUG { 19:27:51 DEBUG "metric": { 19:27:51 DEBUG "instance": "10.142.0.107:9100" 19:27:51 DEBUG }, 19:27:51 DEBUG "value": [ 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG [...] 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG "instance": "10.142.0.43:9100" 19:27:51 DEBUG }, 19:27:51 DEBUG "value": [ 19:27:51 DEBUG 1713378471, 19:27:51 DEBUG "10429230" 19:27:51 DEBUG ] 19:27:51 DEBUG } 19:27:51 DEBUG ] 19:27:51 DEBUG } 19:27:51 DEBUG } 19:27:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378471" 19:27:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378471" 19:27:51 INFO [http_cmd]: http status code OK 19:27:51 DEBUG --- status code --- 19:27:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:51 DEBUG --- http response --- 19:27:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:51 DEBUG ----- output ----- 19:27:51 DEBUG { 19:27:51 DEBUG "status": "success", 19:27:51 DEBUG "data": { 19:27:51 DEBUG "resultType": "vector", 19:27:51 DEBUG "result": [ 19:27:51 DEBUG { 19:27:51 DEBUG "metric": { 19:27:51 DEBUG "pod": "ds-idrepo-0" 19:27:51 DEBUG }, 19:27:51 DEBUG "value": [ 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG [...] 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG "pod": "ds-idrepo-2" 19:27:51 DEBUG }, 19:27:51 DEBUG "value": [ 19:27:51 DEBUG 1713378471, 19:27:51 DEBUG "0" 19:27:51 DEBUG ] 19:27:51 DEBUG } 19:27:51 DEBUG ] 19:27:51 DEBUG } 19:27:51 DEBUG } 19:27:51 INFO [http_cmd]: http status code OK 19:27:51 DEBUG --- status code --- 19:27:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:51 DEBUG --- http response --- 19:27:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:51 DEBUG ----- output ----- 19:27:51 DEBUG { 19:27:51 DEBUG "status": "success", 19:27:51 DEBUG "data": { 19:27:51 DEBUG "resultType": "vector", 19:27:51 DEBUG "result": [ 19:27:51 DEBUG { 19:27:51 DEBUG "metric": { 19:27:51 DEBUG "instance": "10.142.0.107:9100" 19:27:51 DEBUG }, 19:27:51 DEBUG "value": [ 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG [...] 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG 19:27:51 DEBUG "instance": "10.142.0.43:9100" 19:27:51 DEBUG }, 19:27:51 DEBUG "value": [ 19:27:51 DEBUG 1713378471, 19:27:51 DEBUG "10422663.1" 19:27:51 DEBUG ] 19:27:51 DEBUG } 19:27:51 DEBUG ] 19:27:51 DEBUG } 19:27:51 DEBUG } 19:27:51 INFO 19:27:51 INFO [loop_until]: kubectl --namespace=xlou top pods 19:27:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:27:51 INFO [loop_until]: OK (rc = 0) 19:27:51 DEBUG --- stdout --- 19:27:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 6641m 8311Mi am-6844b8c4b7-n64np 5807m 7361Mi am-6844b8c4b7-pzj6t 4629m 7398Mi ds-cts-0 5144m 4769Mi ds-cts-1 4859m 4761Mi ds-cts-2 5658m 4794Mi ds-idrepo-0 9442m 4222Mi ds-idrepo-1 175m 4367Mi ds-idrepo-2 200m 4138Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 6m 950Mi idm-b85f9d978-wnhxb 12m 981Mi lodemon-9886f5b6-dt5fm 1m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 5485m 15Mi 19:27:51 DEBUG --- stderr --- 19:27:51 DEBUG 19:27:51 INFO 19:27:51 INFO [loop_until]: kubectl --namespace=xlou top node 19:27:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:27:51 INFO [loop_until]: OK (rc = 0) 19:27:51 DEBUG --- stdout --- 19:27:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 82m 0% 5493Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 264m 0% 6543Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 4319m 14% 9948Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 3753m 12% 10247Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 5275m 17% 6893Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 137m 0% 4941Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 6076m 20% 6566Mi 5% gke-xlou-cdm-ds-0f601751-cf7b 18196m 60% 11373Mi 10% gke-xlou-cdm-ds-0f601751-hmm7 213m 0% 5694Mi 5% gke-xlou-cdm-ds-0f601751-j488 49m 0% 1671Mi 1% gke-xlou-cdm-ds-0f601751-mc5d 5363m 17% 7876Mi 7% gke-xlou-cdm-ds-0f601751-nj6q 80m 0% 2177Mi 1% gke-xlou-cdm-ds-0f601751-nmxm 5641m 18% 6363Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 5860m 36% 3925Mi 6% 19:27:51 DEBUG --- stderr --- 19:27:51 DEBUG 19:28:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378530" 19:28:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378530" 19:28:50 INFO [http_cmd]: http status code OK 19:28:50 INFO [http_cmd]: http status code OK 19:28:50 DEBUG --- status code --- 19:28:50 DEBUG --- status code --- 19:28:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:50 DEBUG --- http response --- 19:28:50 DEBUG --- http response --- 19:28:50 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:28:50 DEBUG ----- output ----- 19:28:50 DEBUG { 19:28:50 DEBUG "status": "success", 19:28:50 DEBUG "data": { 19:28:50 DEBUG "resultType": "vector", 19:28:50 DEBUG "result": [ 19:28:50 DEBUG { 19:28:50 DEBUG "metric": { 19:28:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG [...] 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 1713378530, 19:28:50 DEBUG "7894385.206282664" 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG } 19:28:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713378530" 19:28:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378530" 19:28:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378530" 19:28:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378530" 19:28:50 INFO [http_cmd]: http status code OK 19:28:50 DEBUG --- status code --- 19:28:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:50 DEBUG --- http response --- 19:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:28:50 DEBUG ----- output ----- 19:28:50 DEBUG { 19:28:50 DEBUG "status": "success", 19:28:50 DEBUG "data": { 19:28:50 DEBUG "resultType": "vector", 19:28:50 DEBUG "result": [ 19:28:50 DEBUG { 19:28:50 DEBUG "metric": { 19:28:50 DEBUG "pod": "ds-idrepo-0" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG [...] 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG "pod": "ds-cts-2" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 1713378530, 19:28:50 DEBUG "0" 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG } 19:28:50 INFO [http_cmd]: http status code OK 19:28:50 DEBUG --- status code --- 19:28:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:50 DEBUG --- http response --- 19:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:28:50 DEBUG ----- output ----- 19:28:50 DEBUG { 19:28:50 DEBUG "status": "success", 19:28:50 DEBUG "data": { 19:28:50 DEBUG "resultType": "vector", 19:28:50 DEBUG "result": [ 19:28:50 DEBUG { 19:28:50 DEBUG "metric": { 19:28:50 DEBUG "pod": "ds-cts-0" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG [...] 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG "pod": "ds-cts-2" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 1713378530, 19:28:50 DEBUG "0.0001" 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG } 19:28:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378530" 19:28:50 INFO [http_cmd]: http status code OK 19:28:50 DEBUG --- status code --- 19:28:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:50 DEBUG --- http response --- 19:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:50 DEBUG ----- output ----- 19:28:50 DEBUG { 19:28:50 DEBUG "status": "success", 19:28:50 DEBUG "data": { 19:28:50 DEBUG "resultType": "vector", 19:28:50 DEBUG "result": [ 19:28:50 DEBUG { 19:28:50 DEBUG "metric": { 19:28:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG [...] 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 1713378530, 19:28:50 DEBUG "0.06666666666666667" 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG } 19:28:50 INFO [http_cmd]: http status code OK 19:28:50 DEBUG --- status code --- 19:28:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:50 DEBUG --- http response --- 19:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:50 DEBUG ----- output ----- 19:28:50 DEBUG { 19:28:50 DEBUG "status": "success", 19:28:50 DEBUG "data": { 19:28:50 DEBUG "resultType": "vector", 19:28:50 DEBUG "result": [ 19:28:50 DEBUG { 19:28:50 DEBUG "metric": { 19:28:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG [...] 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 1713378530, 19:28:50 DEBUG "0" 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG } 19:28:50 INFO [http_cmd]: http status code OK 19:28:50 DEBUG --- status code --- 19:28:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:50 DEBUG --- http response --- 19:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:50 DEBUG ----- output ----- 19:28:50 DEBUG { 19:28:50 DEBUG "status": "success", 19:28:50 DEBUG "data": { 19:28:50 DEBUG "resultType": "vector", 19:28:50 DEBUG "result": [ 19:28:50 DEBUG { 19:28:50 DEBUG "metric": { 19:28:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG [...] 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 1713378530, 19:28:50 DEBUG "145636" 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG } 19:28:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378530" 19:28:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378530" 19:28:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378530" 19:28:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378530" 19:28:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378530" 19:28:50 INFO [http_cmd]: http status code OK 19:28:50 DEBUG --- status code --- 19:28:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:50 DEBUG --- http response --- 19:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:50 DEBUG ----- output ----- 19:28:50 DEBUG { 19:28:50 DEBUG "status": "success", 19:28:50 DEBUG "data": { 19:28:50 DEBUG "resultType": "vector", 19:28:50 DEBUG "result": [ 19:28:50 DEBUG { 19:28:50 DEBUG "metric": { 19:28:50 DEBUG "pod": "ds-idrepo-0" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG [...] 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG "pod": "ds-idrepo-2" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 1713378530, 19:28:50 DEBUG "0" 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG } 19:28:50 INFO [http_cmd]: http status code OK 19:28:50 DEBUG --- status code --- 19:28:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:50 DEBUG --- http response --- 19:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:50 DEBUG ----- output ----- 19:28:50 DEBUG { 19:28:50 DEBUG "status": "success", 19:28:50 DEBUG "data": { 19:28:50 DEBUG "resultType": "vector", 19:28:50 DEBUG "result": [ 19:28:50 DEBUG { 19:28:50 DEBUG "metric": { 19:28:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG [...] 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 1713378530, 19:28:50 DEBUG "1417.1666666666667" 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG } 19:28:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713378530" 19:28:50 INFO [http_cmd]: http status code OK 19:28:50 INFO [http_cmd]: http status code OK 19:28:50 DEBUG --- status code --- 19:28:50 DEBUG --- status code --- 19:28:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:50 DEBUG --- http response --- 19:28:50 DEBUG --- http response --- 19:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:50 DEBUG ----- output ----- 19:28:50 DEBUG { 19:28:50 DEBUG "status": "success", 19:28:50 DEBUG "data": { 19:28:50 DEBUG "resultType": "vector", 19:28:50 DEBUG "result": [ 19:28:50 DEBUG { 19:28:50 DEBUG "metric": { 19:28:50 DEBUG "instance": "10.142.0.107:9100" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG [...] 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG "instance": "10.142.0.43:9100" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 1713378530, 19:28:50 DEBUG "0.013799999999999221" 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG } 19:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:28:50 DEBUG ----- output ----- 19:28:50 DEBUG { 19:28:50 DEBUG "status": "success", 19:28:50 DEBUG "data": { 19:28:50 DEBUG "resultType": "vector", 19:28:50 DEBUG "result": [ 19:28:50 DEBUG { 19:28:50 DEBUG "metric": { 19:28:50 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG [...] 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 1713378530, 19:28:50 DEBUG "1.5201566262426205" 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG } 19:28:50 INFO [http_cmd]: http status code OK 19:28:50 DEBUG --- status code --- 19:28:50 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:50 DEBUG --- http response --- 19:28:50 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:50 DEBUG ----- output ----- 19:28:50 DEBUG { 19:28:50 DEBUG "status": "success", 19:28:50 DEBUG "data": { 19:28:50 DEBUG "resultType": "vector", 19:28:50 DEBUG "result": [ 19:28:50 DEBUG { 19:28:50 DEBUG "metric": { 19:28:50 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG [...] 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG 19:28:50 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:28:50 DEBUG }, 19:28:50 DEBUG "value": [ 19:28:50 DEBUG 1713378530, 19:28:50 DEBUG "0.06666666666666667" 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG ] 19:28:50 DEBUG } 19:28:50 DEBUG } 19:28:50 ERROR [http_cmd]: ERROR 19:28:50 ERROR --- status code --- 19:28:50 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:28:50 ERROR --- http response --- 19:28:50 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:28:50 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:28:50 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378530" 19:28:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378531" 19:28:51 INFO [http_cmd]: http status code OK 19:28:51 DEBUG --- status code --- 19:28:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:51 DEBUG --- http response --- 19:28:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:28:51 DEBUG ----- output ----- 19:28:51 DEBUG { 19:28:51 DEBUG "status": "success", 19:28:51 DEBUG "data": { 19:28:51 DEBUG "resultType": "vector", 19:28:51 DEBUG "result": [ 19:28:51 DEBUG { 19:28:51 DEBUG "metric": { 19:28:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG [...] 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 1713378530, 19:28:51 DEBUG "9823.875342225601" 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378531" 19:28:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713378531" 19:28:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713378531" 19:28:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713378531" 19:28:51 INFO [http_cmd]: http status code OK 19:28:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378531" 19:28:51 DEBUG --- status code --- 19:28:51 INFO [http_cmd]: http status code OK 19:28:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:51 DEBUG --- status code --- 19:28:51 INFO [http_cmd]: http status code OK 19:28:51 INFO [http_cmd]: http status code OK 19:28:51 INFO [http_cmd]: http status code OK 19:28:51 DEBUG --- http response --- 19:28:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:51 DEBUG --- status code --- 19:28:51 DEBUG --- status code --- 19:28:51 DEBUG --- status code --- 19:28:51 DEBUG --- http response --- 19:28:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:51 DEBUG ----- output ----- 19:28:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:51 DEBUG --- http response --- 19:28:51 DEBUG --- http response --- 19:28:51 DEBUG --- http response --- 19:28:51 DEBUG { 19:28:51 DEBUG ----- output ----- 19:28:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:51 DEBUG "status": "success", 19:28:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:51 DEBUG { 19:28:51 DEBUG ----- output ----- 19:28:51 DEBUG ----- output ----- 19:28:51 DEBUG "data": { 19:28:51 DEBUG ----- output ----- 19:28:51 DEBUG "status": "success", 19:28:51 DEBUG { 19:28:51 DEBUG { 19:28:51 DEBUG "resultType": "vector", 19:28:51 DEBUG { 19:28:51 DEBUG "data": { 19:28:51 DEBUG "status": "success", 19:28:51 DEBUG "status": "success", 19:28:51 DEBUG "result": [ 19:28:51 DEBUG "status": "success", 19:28:51 DEBUG "resultType": "vector", 19:28:51 DEBUG "data": { 19:28:51 DEBUG "data": { 19:28:51 DEBUG { 19:28:51 DEBUG "data": { 19:28:51 DEBUG "result": [ 19:28:51 DEBUG "resultType": "vector", 19:28:51 DEBUG "resultType": "vector", 19:28:51 DEBUG "metric": { 19:28:51 DEBUG "resultType": "vector", 19:28:51 DEBUG { 19:28:51 DEBUG "result": [ 19:28:51 DEBUG "result": [ 19:28:51 DEBUG "instance": "10.142.0.107:9100" 19:28:51 DEBUG "result": [ 19:28:51 DEBUG "metric": { 19:28:51 DEBUG { 19:28:51 DEBUG { 19:28:51 DEBUG }, 19:28:51 DEBUG { 19:28:51 DEBUG "instance": "10.142.0.107:9100" 19:28:51 DEBUG "metric": { 19:28:51 DEBUG "metric": { 19:28:51 DEBUG "value": [ 19:28:51 DEBUG "metric": { 19:28:51 DEBUG }, 19:28:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:28:51 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:28:51 DEBUG 19:28:51 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:28:51 DEBUG "value": [ 19:28:51 DEBUG }, 19:28:51 DEBUG }, 19:28:51 DEBUG 19:28:51 DEBUG }, 19:28:51 DEBUG 19:28:51 DEBUG "value": [ 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG [...] 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG [...] 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG [...] 19:28:51 DEBUG [...] 19:28:51 DEBUG 19:28:51 DEBUG [...] 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG "instance": "10.142.0.43:9100" 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG }, 19:28:51 DEBUG 19:28:51 DEBUG "instance": "10.142.0.43:9100" 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 19:28:51 DEBUG }, 19:28:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:28:51 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:28:51 DEBUG 1713378531, 19:28:51 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:28:51 DEBUG "value": [ 19:28:51 DEBUG }, 19:28:51 DEBUG }, 19:28:51 DEBUG "0.010000000000000023" 19:28:51 DEBUG }, 19:28:51 DEBUG 1713378531, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG "value": [ 19:28:51 DEBUG ] 19:28:51 DEBUG "value": [ 19:28:51 DEBUG "44041585664" 19:28:51 DEBUG 1713378531, 19:28:51 DEBUG 1713378531, 19:28:51 DEBUG } 19:28:51 DEBUG 1713378531, 19:28:51 DEBUG ] 19:28:51 INFO [http_cmd]: http status code OK 19:28:51 DEBUG "0.7917420927653945" 19:28:51 DEBUG "252563456" 19:28:51 DEBUG ] 19:28:51 DEBUG "0.006" 19:28:51 DEBUG } 19:28:51 DEBUG --- status code --- 19:28:51 DEBUG ] 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG ] 19:28:51 DEBUG ] 19:28:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 DEBUG --- http response --- 19:28:51 DEBUG ] 19:28:51 DEBUG ] 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 DEBUG ----- output ----- 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 DEBUG { 19:28:51 DEBUG "status": "success", 19:28:51 DEBUG "data": { 19:28:51 DEBUG "resultType": "vector", 19:28:51 DEBUG "result": [ 19:28:51 DEBUG { 19:28:51 DEBUG "metric": { 19:28:51 DEBUG "instance": "10.142.0.107:9100" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG [...] 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG "instance": "10.142.0.43:9100" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 1713378531, 19:28:51 DEBUG "8256316.633333333" 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378531" 19:28:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378531" 19:28:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378531" 19:28:51 INFO [http_cmd]: http status code OK 19:28:51 DEBUG --- status code --- 19:28:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:51 DEBUG --- http response --- 19:28:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:28:51 DEBUG ----- output ----- 19:28:51 DEBUG { 19:28:51 DEBUG "status": "success", 19:28:51 DEBUG "data": { 19:28:51 DEBUG "resultType": "vector", 19:28:51 DEBUG "result": [ 19:28:51 DEBUG { 19:28:51 DEBUG "metric": { 19:28:51 DEBUG "pod": "ds-cts-0" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG [...] 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG "pod": "ds-cts-2" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 1713378531, 19:28:51 DEBUG "0" 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 INFO [http_cmd]: http status code OK 19:28:51 DEBUG --- status code --- 19:28:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:51 DEBUG --- http response --- 19:28:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:28:51 DEBUG ----- output ----- 19:28:51 DEBUG { 19:28:51 DEBUG "status": "success", 19:28:51 DEBUG "data": { 19:28:51 DEBUG "resultType": "vector", 19:28:51 DEBUG "result": [ 19:28:51 DEBUG { 19:28:51 DEBUG "metric": { 19:28:51 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG [...] 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 1713378531, 19:28:51 DEBUG "7637062.708105829" 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378531" 19:28:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378531" 19:28:51 INFO [http_cmd]: http status code OK 19:28:51 DEBUG --- status code --- 19:28:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:51 DEBUG --- http response --- 19:28:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:28:51 DEBUG ----- output ----- 19:28:51 DEBUG { 19:28:51 DEBUG "status": "success", 19:28:51 DEBUG "data": { 19:28:51 DEBUG "resultType": "vector", 19:28:51 DEBUG "result": [ 19:28:51 DEBUG { 19:28:51 DEBUG "metric": { 19:28:51 DEBUG "pod": "ds-cts-0" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG [...] 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG "pod": "ds-cts-2" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 1713378531, 19:28:51 DEBUG "0" 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 INFO [http_cmd]: http status code OK 19:28:51 DEBUG --- status code --- 19:28:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:51 DEBUG --- http response --- 19:28:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:51 DEBUG ----- output ----- 19:28:51 DEBUG { 19:28:51 DEBUG "status": "success", 19:28:51 DEBUG "data": { 19:28:51 DEBUG "resultType": "vector", 19:28:51 DEBUG "result": [ 19:28:51 DEBUG { 19:28:51 DEBUG "metric": { 19:28:51 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG [...] 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 1713378531, 19:28:51 DEBUG "0.03333333333333333" 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 INFO [http_cmd]: http status code OK 19:28:51 DEBUG --- status code --- 19:28:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:51 DEBUG --- http response --- 19:28:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:51 DEBUG ----- output ----- 19:28:51 DEBUG { 19:28:51 DEBUG "status": "success", 19:28:51 DEBUG "data": { 19:28:51 DEBUG "resultType": "vector", 19:28:51 DEBUG "result": [ 19:28:51 DEBUG { 19:28:51 DEBUG "metric": { 19:28:51 DEBUG "pod": "ds-idrepo-0" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG [...] 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG "pod": "ds-idrepo-2" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 1713378531, 19:28:51 DEBUG "0" 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713378531" 19:28:51 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713378531" 19:28:51 INFO [http_cmd]: http status code OK 19:28:51 DEBUG --- status code --- 19:28:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:51 DEBUG --- http response --- 19:28:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:51 DEBUG ----- output ----- 19:28:51 DEBUG { 19:28:51 DEBUG "status": "success", 19:28:51 DEBUG "data": { 19:28:51 DEBUG "resultType": "vector", 19:28:51 DEBUG "result": [ 19:28:51 DEBUG { 19:28:51 DEBUG "metric": { 19:28:51 DEBUG "pod": "ds-idrepo-0" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG [...] 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG "pod": "ds-idrepo-2" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 1713378531, 19:28:51 DEBUG "0" 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 INFO [http_cmd]: http status code OK 19:28:51 DEBUG --- status code --- 19:28:51 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:51 DEBUG --- http response --- 19:28:51 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:51 DEBUG ----- output ----- 19:28:51 DEBUG { 19:28:51 DEBUG "status": "success", 19:28:51 DEBUG "data": { 19:28:51 DEBUG "resultType": "vector", 19:28:51 DEBUG "result": [ 19:28:51 DEBUG { 19:28:51 DEBUG "metric": { 19:28:51 DEBUG "instance": "10.142.0.107:9100" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG [...] 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG 19:28:51 DEBUG "instance": "10.142.0.43:9100" 19:28:51 DEBUG }, 19:28:51 DEBUG "value": [ 19:28:51 DEBUG 1713378531, 19:28:51 DEBUG "8250177.133333333" 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG ] 19:28:51 DEBUG } 19:28:51 DEBUG } 19:28:51 INFO 19:28:51 INFO [loop_until]: kubectl --namespace=xlou top pods 19:28:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:28:51 INFO [loop_until]: OK (rc = 0) 19:28:51 DEBUG --- stdout --- 19:28:51 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 1428m 8256Mi am-6844b8c4b7-n64np 1252m 7322Mi am-6844b8c4b7-pzj6t 1181m 7377Mi ds-cts-0 3175m 4782Mi ds-cts-1 3024m 4751Mi ds-cts-2 3199m 4795Mi ds-idrepo-0 1524m 4246Mi ds-idrepo-1 391m 4370Mi ds-idrepo-2 262m 4138Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 11m 951Mi idm-b85f9d978-wnhxb 8m 981Mi lodemon-9886f5b6-dt5fm 10m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 587m 487Mi 19:28:51 DEBUG --- stderr --- 19:28:51 DEBUG 19:28:51 INFO 19:28:51 INFO [loop_until]: kubectl --namespace=xlou top node 19:28:51 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:28:51 INFO [loop_until]: OK (rc = 0) 19:28:51 DEBUG --- stdout --- 19:28:51 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 84m 0% 5492Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 354m 1% 6487Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1283m 4% 9959Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 1333m 4% 10271Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 3296m 11% 6927Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 4942Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 1696m 5% 6554Mi 5% gke-xlou-cdm-ds-0f601751-cf7b 1684m 5% 11243Mi 10% gke-xlou-cdm-ds-0f601751-hmm7 446m 1% 5659Mi 5% gke-xlou-cdm-ds-0f601751-j488 53m 0% 1688Mi 1% gke-xlou-cdm-ds-0f601751-mc5d 3133m 10% 7875Mi 7% gke-xlou-cdm-ds-0f601751-nj6q 81m 0% 2182Mi 1% gke-xlou-cdm-ds-0f601751-nmxm 3157m 10% 6373Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 758m 4% 4262Mi 7% 19:28:51 DEBUG --- stderr --- 19:28:51 DEBUG 19:29:12 INFO Finished: True 19:29:12 INFO Waiting for threads to register finish flag 19:29:51 INFO Done. Have a nice day! :) 127.0.0.1 - - [17/Apr/2024 19:29:51] "GET /monitoring/stop HTTP/1.1" 200 - 19:29:54 WARNING File /tmp/lodemon_data-20240417_191547/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6471', ...) 127.0.0.1 - - [17/Apr/2024 19:29:58] "GET /monitoring/process HTTP/1.1" 200 -