==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-5cd7976d78-6npxn Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Wed, 17 Apr 2024 18:44:03 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=5cd7976d78 skaffold.dev/run-id=41d1b2f9-7d46-4c23-923b-2b74efddcc52 Annotations: Status: Running IP: 10.106.46.163 IPs: IP: 10.106.46.163 Controlled By: ReplicaSet/lodemon-5cd7976d78 Containers: lodemon: Container ID: containerd://203dc03c8255fe672b4ce881b09bf9bea3551cc8e90567dba7ae3123e0b35f60 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:2febaeab396557db3e0fc07c0501fa786a35d3473828be1a84a3d1610beec5de Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Wed, 17 Apr 2024 18:44:20 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fk2zq (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-fk2zq: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 15m default-scheduler Successfully assigned xlou/lodemon-5cd7976d78-6npxn to gke-xlou-cdm-ds-0f601751-mc5d Normal Pulling 15m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 15m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 15.407078547s (15.407092s including waiting) Normal Created 15m kubelet Created container lodemon Normal Started 15m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.46.163:8080 Press CTRL+C to quit 19:45:21 INFO 19:45:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:21 INFO [loop_until]: OK (rc = 0) 19:45:21 DEBUG --- stdout --- 19:45:21 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:21 DEBUG --- stderr --- 19:45:21 DEBUG 19:45:21 INFO 19:45:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:21 INFO [loop_until]: OK (rc = 0) 19:45:21 DEBUG --- stdout --- 19:45:21 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:21 DEBUG --- stderr --- 19:45:21 DEBUG 19:45:21 INFO 19:45:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:21 INFO [loop_until]: OK (rc = 0) 19:45:21 DEBUG --- stdout --- 19:45:21 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:21 DEBUG --- stderr --- 19:45:21 DEBUG 19:45:21 INFO 19:45:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:21 INFO [loop_until]: OK (rc = 0) 19:45:21 DEBUG --- stdout --- 19:45:21 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:21 DEBUG --- stderr --- 19:45:21 DEBUG 19:45:21 INFO 19:45:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:21 INFO [loop_until]: OK (rc = 0) 19:45:21 DEBUG --- stdout --- 19:45:21 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:21 DEBUG --- stderr --- 19:45:21 DEBUG 19:45:21 INFO 19:45:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:22 INFO [loop_until]: OK (rc = 0) 19:45:22 DEBUG --- stdout --- 19:45:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:22 DEBUG --- stderr --- 19:45:22 DEBUG 19:45:22 INFO 19:45:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:22 INFO [loop_until]: OK (rc = 0) 19:45:22 DEBUG --- stdout --- 19:45:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:22 DEBUG --- stderr --- 19:45:22 DEBUG 19:45:22 INFO 19:45:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:22 INFO [loop_until]: OK (rc = 0) 19:45:22 DEBUG --- stdout --- 19:45:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:22 DEBUG --- stderr --- 19:45:22 DEBUG 19:45:22 INFO 19:45:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:22 INFO [loop_until]: OK (rc = 0) 19:45:22 DEBUG --- stdout --- 19:45:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:22 DEBUG --- stderr --- 19:45:22 DEBUG 19:45:22 INFO 19:45:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:22 INFO [loop_until]: OK (rc = 0) 19:45:22 DEBUG --- stdout --- 19:45:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:22 DEBUG --- stderr --- 19:45:22 DEBUG 19:45:22 INFO 19:45:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:22 INFO [loop_until]: OK (rc = 0) 19:45:22 DEBUG --- stdout --- 19:45:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:22 DEBUG --- stderr --- 19:45:22 DEBUG 19:45:22 INFO 19:45:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:22 INFO [loop_until]: OK (rc = 0) 19:45:22 DEBUG --- stdout --- 19:45:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:22 DEBUG --- stderr --- 19:45:22 DEBUG 19:45:22 INFO 19:45:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:22 INFO [loop_until]: OK (rc = 0) 19:45:22 DEBUG --- stdout --- 19:45:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:22 DEBUG --- stderr --- 19:45:22 DEBUG 19:45:23 INFO 19:45:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:23 INFO [loop_until]: OK (rc = 0) 19:45:23 DEBUG --- stdout --- 19:45:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:23 DEBUG --- stderr --- 19:45:23 DEBUG 19:45:23 INFO 19:45:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:23 INFO [loop_until]: OK (rc = 0) 19:45:23 DEBUG --- stdout --- 19:45:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:23 DEBUG --- stderr --- 19:45:23 DEBUG 19:45:23 INFO 19:45:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:23 INFO [loop_until]: OK (rc = 0) 19:45:23 DEBUG --- stdout --- 19:45:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:23 DEBUG --- stderr --- 19:45:23 DEBUG 19:45:23 INFO 19:45:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:23 INFO [loop_until]: OK (rc = 0) 19:45:23 DEBUG --- stdout --- 19:45:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:23 DEBUG --- stderr --- 19:45:23 DEBUG 19:45:23 INFO 19:45:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:23 INFO [loop_until]: OK (rc = 0) 19:45:23 DEBUG --- stdout --- 19:45:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:23 DEBUG --- stderr --- 19:45:23 DEBUG 19:45:23 INFO 19:45:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:23 INFO [loop_until]: OK (rc = 0) 19:45:23 DEBUG --- stdout --- 19:45:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:23 DEBUG --- stderr --- 19:45:23 DEBUG 19:45:23 INFO 19:45:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:23 INFO [loop_until]: OK (rc = 0) 19:45:23 DEBUG --- stdout --- 19:45:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:23 DEBUG --- stderr --- 19:45:23 DEBUG 19:45:23 INFO 19:45:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:24 INFO [loop_until]: OK (rc = 0) 19:45:24 DEBUG --- stdout --- 19:45:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:24 DEBUG --- stderr --- 19:45:24 DEBUG 19:45:24 INFO 19:45:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:24 INFO [loop_until]: OK (rc = 0) 19:45:24 DEBUG --- stdout --- 19:45:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:24 DEBUG --- stderr --- 19:45:24 DEBUG 19:45:24 INFO 19:45:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:24 INFO [loop_until]: OK (rc = 0) 19:45:24 DEBUG --- stdout --- 19:45:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:24 DEBUG --- stderr --- 19:45:24 DEBUG 19:45:24 INFO 19:45:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:24 INFO [loop_until]: OK (rc = 0) 19:45:24 DEBUG --- stdout --- 19:45:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:24 DEBUG --- stderr --- 19:45:24 DEBUG 19:45:24 INFO 19:45:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:24 INFO [loop_until]: OK (rc = 0) 19:45:24 DEBUG --- stdout --- 19:45:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:24 DEBUG --- stderr --- 19:45:24 DEBUG 19:45:24 INFO 19:45:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:24 INFO [loop_until]: OK (rc = 0) 19:45:24 DEBUG --- stdout --- 19:45:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:24 DEBUG --- stderr --- 19:45:24 DEBUG 19:45:24 INFO 19:45:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 19:45:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:24 INFO [loop_until]: OK (rc = 0) 19:45:24 DEBUG --- stdout --- 19:45:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 19:45:24 DEBUG --- stderr --- 19:45:24 DEBUG 19:45:24 INFO Initializing monitoring instance threads 19:45:24 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 19:45:24 INFO Starting instance threads 19:45:24 INFO 19:45:24 INFO Thread started 19:45:24 INFO [loop_until]: kubectl --namespace=xlou top node 19:45:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:24 INFO 19:45:24 INFO Thread started 19:45:24 INFO [loop_until]: kubectl --namespace=xlou top pods 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG --- status code --- 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 DEBUG --- status code --- 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG --- http response --- 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 INFO Thread started 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:45:24 DEBUG --- http response --- 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 DEBUG --- http response --- 19:45:24 INFO Thread started 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:45:24 INFO Thread started 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:45:24 DEBUG { 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG { 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 DEBUG ----- output ----- 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 INFO [loop_until]: OK (rc = 0) 19:45:24 DEBUG { 19:45:24 INFO Thread started 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG { 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG { 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG { 19:45:24 DEBUG { 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG { 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG "data": { 19:45:24 DEBUG "status": "success", 19:45:24 INFO Thread started 19:45:24 DEBUG { 19:45:24 DEBUG { 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG { 19:45:24 DEBUG { 19:45:24 DEBUG { 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG "data": { 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG "data": { 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "data": { 19:45:24 DEBUG "status": "success", 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713379524" 19:45:24 DEBUG "status": "success", 19:45:24 INFO Thread started 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:45:24 DEBUG "data": { 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG "data": { 19:45:24 DEBUG "resultType": "vector", 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 DEBUG "data": { 19:45:24 DEBUG "data": { 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG "result": [ 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "data": { 19:45:24 DEBUG "data": { 19:45:24 DEBUG --- stdout --- 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG { 19:45:24 DEBUG ----- output ----- 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "data": { 19:45:24 DEBUG "data": { 19:45:24 DEBUG "data": { 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "result": [ 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "result": [ 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG { 19:45:24 DEBUG "result": [ 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG { 19:45:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 4m 8236Mi am-6844b8c4b7-n64np 2m 7773Mi am-6844b8c4b7-pzj6t 3m 7394Mi ds-cts-0 7m 916Mi ds-cts-1 9m 843Mi ds-cts-2 7m 963Mi ds-idrepo-0 17m 892Mi ds-idrepo-1 15m 783Mi ds-idrepo-2 15m 759Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 960Mi idm-b85f9d978-wnhxb 4m 992Mi lodemon-5cd7976d78-6npxn 45m 55Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1m 254Mi 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG { 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379524" 19:45:24 DEBUG ----- output ----- 19:45:24 INFO Thread started 19:45:24 DEBUG { 19:45:24 DEBUG "result": [ 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "result": [ 19:45:24 DEBUG { 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG "result": [ 19:45:24 DEBUG "result": [ 19:45:24 DEBUG { 19:45:24 DEBUG ----- output ----- 19:45:24 INFO [loop_until]: OK (rc = 0) 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG "metric": { 19:45:24 DEBUG { 19:45:24 DEBUG "result": [ 19:45:24 DEBUG "result": [ 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG --- stderr --- 19:45:24 DEBUG "data": { 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG { 19:45:24 DEBUG "status": "success", 19:45:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379524" 19:45:24 INFO Thread started 19:45:24 DEBUG { 19:45:24 DEBUG "result": [ 19:45:24 DEBUG "result": [ 19:45:24 DEBUG "result": [ 19:45:24 DEBUG { 19:45:24 DEBUG "metric": { 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG { 19:45:24 DEBUG { 19:45:24 DEBUG "metric": { 19:45:24 DEBUG { 19:45:24 DEBUG --- stdout --- 19:45:24 ERROR [http_cmd]: ERROR 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:45:24 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:45:24 DEBUG "metric": { 19:45:24 DEBUG { 19:45:24 DEBUG { 19:45:24 DEBUG "data": { 19:45:24 DEBUG 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "data": { 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG "data": { 19:45:24 INFO All threads has been started 127.0.0.1 - - [17/Apr/2024 19:45:24] "GET /monitoring/start HTTP/1.1" 200 - 19:45:24 DEBUG "metric": { 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 DEBUG { 19:45:24 DEBUG { 19:45:24 DEBUG { 19:45:24 DEBUG "metric": { 19:45:24 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG "metric": { 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:45:24 DEBUG "metric": { 19:45:24 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 5549Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 79m 0% 3679Mi 3% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 10380Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 10286Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 61m 0% 3850Mi 3% gke-xlou-cdm-default-pool-1b0769c8-wpwl 162m 0% 5003Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2249Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 69m 0% 11128Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 51m 0% 1482Mi 1% gke-xlou-cdm-ds-0f601751-j488 71m 0% 3249Mi 2% gke-xlou-cdm-ds-0f601751-mc5d 152m 0% 4735Mi 4% gke-xlou-cdm-ds-0f601751-nj6q 97m 0% 3794Mi 3% gke-xlou-cdm-ds-0f601751-nmxm 63m 0% 3263Mi 2% gke-xlou-cdm-frontend-a8771548-i82n 73m 0% 4071Mi 6% 19:45:24 ERROR --- status code --- 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG }, 19:45:24 DEBUG "instance": "10.142.0.107:9100" 19:45:24 DEBUG "metric": { 19:45:24 DEBUG "metric": { 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "result": [ 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "data": { 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "instance": "10.142.0.107:9100" 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG "metric": { 19:45:24 DEBUG "metric": { 19:45:24 DEBUG "metric": { 19:45:24 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:45:24 DEBUG }, 19:45:24 DEBUG "instance": "10.142.0.107:9100" 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 DEBUG }, 19:45:24 DEBUG "data": { 19:45:24 DEBUG --- stderr --- 19:45:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:45:24 DEBUG { 19:45:24 DEBUG "value": [ 19:45:24 DEBUG }, 19:45:24 DEBUG "pod": "ds-idrepo-0" 19:45:24 DEBUG "pod": "ds-cts-0" 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG "result": [ 19:45:24 DEBUG { 19:45:24 DEBUG "result": [ 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "result": [ 19:45:24 DEBUG }, 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:45:24 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:45:24 DEBUG "pod": "ds-idrepo-0" 19:45:24 DEBUG }, 19:45:24 DEBUG "value": [ 19:45:24 DEBUG }, 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG { 19:45:24 DEBUG }, 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "resultType": "vector", 19:45:24 ERROR --- http response --- 19:45:24 DEBUG 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG }, 19:45:24 DEBUG }, 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG { 19:45:24 DEBUG "metric": { 19:45:24 DEBUG { 19:45:24 DEBUG "result": [ 19:45:24 DEBUG { 19:45:24 DEBUG "value": [ 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG }, 19:45:24 DEBUG }, 19:45:24 DEBUG }, 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG { 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG "value": [ 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG 19:45:24 INFO [http_cmd]: http status code OK 19:45:24 DEBUG "result": [ 19:45:24 DEBUG "data": { 19:45:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "value": [ 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG "metric": { 19:45:24 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:45:24 DEBUG "metric": { 19:45:24 DEBUG { 19:45:24 DEBUG "metric": { 19:45:24 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG "data": { 19:45:24 DEBUG 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG 19:45:24 DEBUG --- status code --- 19:45:24 DEBUG { 19:45:24 DEBUG "resultType": "vector", 19:45:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "pod": "ds-idrepo-0" 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:45:24 DEBUG }, 19:45:24 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:45:24 DEBUG "metric": { 19:45:24 DEBUG 19:45:24 DEBUG }, 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "data": { 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:45:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:45:24 DEBUG "metric": { 19:45:24 DEBUG "result": [ 19:45:24 DEBUG [...] 19:45:24 DEBUG 19:45:24 DEBUG [...] 19:45:24 DEBUG 19:45:24 DEBUG }, 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG "value": [ 19:45:24 DEBUG }, 19:45:24 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:45:24 DEBUG }, 19:45:24 DEBUG 19:45:24 DEBUG { 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG [...] 19:45:24 DEBUG 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG "result": [ 19:45:24 DEBUG 19:45:24 DEBUG [...] 19:45:24 DEBUG --- http response --- 19:45:24 DEBUG "pod": "ds-cts-0" 19:45:24 DEBUG { 19:45:24 DEBUG "metric": { 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG { 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG [...] 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG [...] 19:45:24 DEBUG "result": [ 19:45:24 DEBUG { 19:45:24 DEBUG [...] 19:45:24 DEBUG 19:45:24 DEBUG }, 19:45:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:45:24 DEBUG ----- output ----- 19:45:24 DEBUG "pod": "ds-cts-0" 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG [...] 19:45:24 DEBUG 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG [...] 19:45:24 DEBUG "data": { 19:45:24 DEBUG [...] 19:45:24 DEBUG [...] 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG { 19:45:24 DEBUG "metric": { 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 19:45:24 DEBUG { 19:45:24 DEBUG "status": "success", 19:45:24 DEBUG [...] 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "data": { 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG [...] 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "metric": { 19:45:24 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG }, 19:45:24 DEBUG "data": { 19:45:24 DEBUG 19:45:24 DEBUG "instance": "10.142.0.43:9100" 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG [...] 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG [...] 19:45:24 DEBUG "result": [ 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "pod": "ds-idrepo-2" 19:45:24 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:45:24 DEBUG 19:45:24 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:45:24 DEBUG }, 19:45:24 DEBUG 19:45:24 DEBUG "pod": "idm-b85f9d978-txpwv" 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "resultType": "vector", 19:45:24 DEBUG 19:45:24 DEBUG }, 19:45:24 DEBUG 19:45:24 DEBUG [...] 19:45:24 DEBUG "result": [ 19:45:24 DEBUG 19:45:24 DEBUG [...] 19:45:24 DEBUG [...] 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG { 19:45:24 DEBUG 19:45:24 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:45:24 DEBUG 19:45:24 DEBUG }, 19:45:24 DEBUG "value": [ 19:45:24 DEBUG }, 19:45:24 DEBUG }, 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "pod": "idm-b85f9d978-txpwv" 19:45:24 DEBUG }, 19:45:24 DEBUG 19:45:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:45:24 DEBUG 19:45:24 DEBUG "result": [ 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "pod": "ds-cts-2" 19:45:24 DEBUG 19:45:24 DEBUG { 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "metric": { 19:45:24 DEBUG 19:45:24 DEBUG }, 19:45:24 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:45:24 DEBUG "instance": "10.142.0.43:9100" 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 19:45:24 DEBUG }, 19:45:24 DEBUG "value": [ 19:45:24 DEBUG [...] 19:45:24 DEBUG }, 19:45:24 DEBUG 19:45:24 DEBUG { 19:45:24 DEBUG "pod": "ds-idrepo-2" 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG }, 19:45:24 DEBUG 19:45:24 DEBUG "metric": { 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "instance": "10.142.0.43:9100" 19:45:24 DEBUG }, 19:45:24 DEBUG "pod": "ds-idrepo-0" 19:45:24 DEBUG }, 19:45:24 DEBUG "value": [ 19:45:24 DEBUG }, 19:45:24 DEBUG }, 19:45:24 DEBUG "0" 19:45:24 DEBUG ] 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG "metric": { 19:45:24 DEBUG }, 19:45:24 DEBUG "13410.833333333334" 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 19:45:24 DEBUG "instance": "10.142.0.107:9100" 19:45:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG } 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG "0" 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG "364359680" 19:45:24 DEBUG "instance": "10.142.0.107:9100" 19:45:24 DEBUG "value": [ 19:45:24 DEBUG ] 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG "pod": "ds-idrepo-2" 19:45:24 DEBUG }, 19:45:24 DEBUG }, 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:45:24 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG "pod": "idm-b85f9d978-txpwv" 19:45:24 DEBUG 19:45:24 DEBUG "0" 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG "0" 19:45:24 DEBUG ] 19:45:24 DEBUG 19:45:24 DEBUG [...] 19:45:24 DEBUG "0" 19:45:24 DEBUG ] 19:45:24 DEBUG 19:45:24 DEBUG [...] 19:45:24 DEBUG ] 19:45:24 DEBUG }, 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG } 19:45:24 DEBUG "0" 19:45:24 DEBUG }, 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "pod": "idm-b85f9d978-txpwv" 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG }, 19:45:24 DEBUG }, 19:45:24 DEBUG "12896.599999999999" 19:45:24 DEBUG }, 19:45:24 DEBUG 19:45:24 DEBUG ] 19:45:24 DEBUG "0" 19:45:24 DEBUG "0.12466666666666668" 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG [...] 19:45:24 DEBUG 19:45:24 DEBUG 19:45:24 DEBUG } 19:45:24 DEBUG "pod": "ds-cts-2" 19:45:24 DEBUG }, 19:45:24 DEBUG } 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 19:45:24 DEBUG ] 19:45:24 DEBUG ] 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 19:45:24 DEBUG }, 19:45:24 DEBUG "0.030583227570075543" 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "value": [ 19:45:24 DEBUG ] 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG 19:45:24 DEBUG ] 19:45:24 DEBUG 19:45:24 DEBUG ] 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG ] 19:45:24 DEBUG "0" 19:45:24 DEBUG 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG 19:45:24 DEBUG "value": [ 19:45:24 DEBUG ] 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG } 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG [...] 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG 19:45:24 DEBUG } 19:45:24 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:45:24 DEBUG } 19:45:24 DEBUG 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG "0" 19:45:24 DEBUG 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG } 19:45:24 DEBUG "0" 19:45:24 DEBUG "0" 19:45:24 DEBUG ] 19:45:24 DEBUG "0.0036499081894960997" 19:45:24 DEBUG 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG 19:45:24 DEBUG }, 19:45:24 DEBUG } 19:45:24 DEBUG 19:45:24 DEBUG "0" 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG [...] 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG [...] 19:45:24 DEBUG "0" 19:45:24 DEBUG ] 19:45:24 DEBUG ] 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "pod": "ds-cts-2" 19:45:24 DEBUG ] 19:45:24 DEBUG ] 19:45:24 DEBUG 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG }, 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG }, 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG ] 19:45:24 DEBUG "pod": "ds-cts-2" 19:45:24 DEBUG } 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "2" 19:45:24 DEBUG "value": [ 19:45:24 DEBUG } 19:45:24 DEBUG 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG } 19:45:24 DEBUG 19:45:24 DEBUG } 19:45:24 DEBUG "instance": "10.142.0.43:9100" 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG }, 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG } 19:45:24 DEBUG "0" 19:45:24 DEBUG "instance": "10.142.0.43:9100" 19:45:24 DEBUG }, 19:45:24 DEBUG } 19:45:24 DEBUG "value": [ 19:45:24 DEBUG "0.008" 19:45:24 DEBUG ] 19:45:24 DEBUG ] 19:45:24 DEBUG }, 19:45:24 DEBUG "value": [ 19:45:24 DEBUG } 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG "value": [ 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG "0" 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG 1713379524, 19:45:24 DEBUG "47152321536" 19:45:24 DEBUG ] 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG "0.14026666666666615" 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG ] 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG } 19:45:24 DEBUG } 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO 19:46:25 INFO [loop_until]: kubectl --namespace=xlou top pods 19:46:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:46:25 INFO 19:46:25 INFO [loop_until]: kubectl --namespace=xlou top node 19:46:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG { 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "data": { 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "result": [ 19:46:25 DEBUG { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG "267.8" 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- http response --- 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713379585" 19:46:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 ERROR [http_cmd]: ERROR 19:46:25 ERROR --- status code --- 19:46:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:46:25 ERROR --- http response --- 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:46:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379585" 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [loop_until]: OK (rc = 0) 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 DEBUG --- status code --- 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 DEBUG --- status code --- 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 DEBUG --- status code --- 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG --- status code --- 19:46:25 INFO [loop_until]: OK (rc = 0) 19:46:25 DEBUG --- status code --- 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG --- status code --- 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG --- stdout --- 19:46:25 DEBUG --- status code --- 19:46:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- stdout --- 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 548m 8272Mi am-6844b8c4b7-n64np 100m 7805Mi am-6844b8c4b7-pzj6t 260m 7418Mi ds-cts-0 122m 929Mi ds-cts-1 450m 887Mi ds-cts-2 258m 1012Mi ds-idrepo-0 208m 934Mi ds-idrepo-1 20m 784Mi ds-idrepo-2 56m 770Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 959Mi idm-b85f9d978-wnhxb 4m 992Mi lodemon-5cd7976d78-6npxn 184m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 1700m 623Mi 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5437Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 167m 0% 3721Mi 3% gke-xlou-cdm-default-pool-1b0769c8-nhlf 628m 2% 11336Mi 10% gke-xlou-cdm-default-pool-1b0769c8-qkns 197m 0% 10299Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 357m 1% 3861Mi 3% gke-xlou-cdm-default-pool-1b0769c8-wpwl 138m 0% 4998Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2245Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 792m 2% 11160Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 50m 0% 1478Mi 1% gke-xlou-cdm-ds-0f601751-j488 122m 0% 3255Mi 2% gke-xlou-cdm-ds-0f601751-mc5d 322m 1% 4724Mi 4% gke-xlou-cdm-ds-0f601751-nj6q 236m 0% 3789Mi 3% gke-xlou-cdm-ds-0f601751-nmxm 136m 0% 3261Mi 2% gke-xlou-cdm-frontend-a8771548-i82n 2277m 14% 4484Mi 7% 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG --- status code --- 19:46:25 INFO [http_cmd]: http status code OK 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG --- stderr --- 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:46:25 DEBUG --- stderr --- 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG --- status code --- 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:46:25 DEBUG 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG { 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG --- http response --- 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG { 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "data": { 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "data": { 19:46:25 DEBUG "data": { 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "data": { 19:46:25 DEBUG "data": { 19:46:25 DEBUG "data": { 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "data": { 19:46:25 DEBUG "data": { 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "data": { 19:46:25 DEBUG "data": { 19:46:25 DEBUG "data": { 19:46:25 DEBUG ----- output ----- 19:46:25 DEBUG { 19:46:25 DEBUG "data": { 19:46:25 DEBUG "data": { 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "data": { 19:46:25 DEBUG "data": { 19:46:25 DEBUG "data": { 19:46:25 DEBUG "data": { 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "data": { 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "data": { 19:46:25 DEBUG "data": { 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "data": { 19:46:25 DEBUG "data": { 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG { 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "result": [ 19:46:25 DEBUG { 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "status": "success", 19:46:25 DEBUG "data": { 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "result": [ 19:46:25 DEBUG { 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "result": [ 19:46:25 DEBUG { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "result": [ 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "result": [ 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG "data": { 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "instance": "10.142.0.107:9100" 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG { 19:46:25 DEBUG { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "resultType": "vector", 19:46:25 DEBUG "result": [ 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:46:25 DEBUG }, 19:46:25 DEBUG { 19:46:25 DEBUG "pod": "ds-idrepo-0" 19:46:25 DEBUG "instance": "10.142.0.107:9100" 19:46:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "pod": "ds-cts-0" 19:46:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:46:25 DEBUG "pod": "ds-idrepo-0" 19:46:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:46:25 DEBUG "result": [ 19:46:25 DEBUG { 19:46:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:46:25 DEBUG "pod": "ds-idrepo-0" 19:46:25 DEBUG }, 19:46:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:46:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:46:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:46:25 DEBUG "pod": "ds-idrepo-0" 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "metric": { 19:46:25 DEBUG }, 19:46:25 DEBUG }, 19:46:25 DEBUG }, 19:46:25 DEBUG "instance": "10.142.0.107:9100" 19:46:25 DEBUG "instance": "10.142.0.107:9100" 19:46:25 DEBUG }, 19:46:25 DEBUG }, 19:46:25 DEBUG "pod": "ds-cts-0" 19:46:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:46:25 DEBUG }, 19:46:25 DEBUG }, 19:46:25 DEBUG }, 19:46:25 DEBUG { 19:46:25 DEBUG "metric": { 19:46:25 DEBUG }, 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG }, 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 19:46:25 DEBUG "instance": "10.142.0.107:9100" 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "value": [ 19:46:25 DEBUG }, 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 19:46:25 DEBUG }, 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "metric": { 19:46:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG "value": [ 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG }, 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG "pod": "ds-cts-0" 19:46:25 DEBUG }, 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG "value": [ 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG [...] 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG [...] 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG [...] 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG "pod": "ds-cts-2" 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG }, 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG [...] 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG "pod": "ds-idrepo-2" 19:46:25 DEBUG 19:46:25 DEBUG }, 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:46:25 DEBUG "pod": "ds-cts-2" 19:46:25 DEBUG "pod": "ds-cts-0" 19:46:25 DEBUG 19:46:25 DEBUG "pod": "ds-idrepo-2" 19:46:25 DEBUG "instance": "10.142.0.43:9100" 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 19:46:25 DEBUG "instance": "10.142.0.43:9100" 19:46:25 DEBUG 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "pod": "ds-cts-2" 19:46:25 DEBUG 19:46:25 DEBUG "pod": "ds-idrepo-2" 19:46:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG }, 19:46:25 DEBUG }, 19:46:25 DEBUG 19:46:25 DEBUG }, 19:46:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:46:25 DEBUG 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG 19:46:25 DEBUG }, 19:46:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG }, 19:46:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 19:46:25 DEBUG 19:46:25 DEBUG }, 19:46:25 DEBUG 19:46:25 DEBUG "366.560305" 19:46:25 DEBUG "instance": "10.142.0.43:9100" 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "instance": "10.142.0.43:9100" 19:46:25 DEBUG "value": [ 19:46:25 DEBUG }, 19:46:25 DEBUG "instance": "10.142.0.43:9100" 19:46:25 DEBUG "0.010999999999999973" 19:46:25 DEBUG 19:46:25 DEBUG "value": [ 19:46:25 DEBUG }, 19:46:25 DEBUG "0" 19:46:25 DEBUG "value": [ 19:46:25 DEBUG }, 19:46:25 DEBUG }, 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG 19:46:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 19:46:25 DEBUG ] 19:46:25 DEBUG }, 19:46:25 DEBUG "0" 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG }, 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG }, 19:46:25 DEBUG ] 19:46:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG ] 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "367.78887499999996" 19:46:25 DEBUG "pod": "ds-cts-2" 19:46:25 DEBUG }, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG } 19:46:25 DEBUG "value": [ 19:46:25 DEBUG ] 19:46:25 DEBUG "10161948.555058334" 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG "0" 19:46:25 DEBUG "0" 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "402.96621666666664" 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG "value": [ 19:46:25 DEBUG } 19:46:25 DEBUG }, 19:46:25 DEBUG "10532101.762365" 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG "0.008" 19:46:25 DEBUG "0" 19:46:25 DEBUG }, 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG "pod": "ds-cts-0" 19:46:25 DEBUG "4005" 19:46:25 DEBUG ] 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG "0" 19:46:25 DEBUG ] 19:46:25 DEBUG ] 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG ] 19:46:25 DEBUG "0" 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG ] 19:46:25 DEBUG "value": [ 19:46:25 DEBUG ] 19:46:25 DEBUG "831045632" 19:46:25 DEBUG } 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG "267.46666666666664" 19:46:25 DEBUG ] 19:46:25 DEBUG ] 19:46:25 DEBUG "value": [ 19:46:25 DEBUG "2.1176512547124915" 19:46:25 DEBUG }, 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG "0" 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG "48361589760" 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG "0.014299999999999594" 19:46:25 DEBUG } 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG ] 19:46:25 DEBUG "value": [ 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG ] 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG "267.8333333333333" 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG ] 19:46:25 DEBUG "0" 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG 1713379585, 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG ] 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG "3780.5557577750974" 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG ] 19:46:25 DEBUG ] 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG ] 19:46:25 DEBUG } 19:46:25 DEBUG } 19:46:25 DEBUG } 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG { 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "data": { 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "result": [ 19:47:25 DEBUG { 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:47:25 DEBUG }, 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:47:25 DEBUG }, 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG "1302.0666666666666" 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713379645" 19:47:25 INFO 19:47:25 INFO [loop_until]: kubectl --namespace=xlou top pods 19:47:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:47:25 INFO 19:47:25 ERROR [http_cmd]: ERROR 19:47:25 INFO [loop_until]: kubectl --namespace=xlou top node 19:47:25 ERROR --- status code --- 19:47:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:47:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:47:25 ERROR --- http response --- 19:47:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:47:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379645" 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379645" 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [loop_until]: OK (rc = 0) 19:47:25 DEBUG --- status code --- 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 DEBUG --- status code --- 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG --- stdout --- 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 DEBUG --- status code --- 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 DEBUG --- status code --- 19:47:25 INFO [loop_until]: OK (rc = 0) 19:47:25 DEBUG --- status code --- 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 8980m 8359Mi am-6844b8c4b7-n64np 5481m 8887Mi am-6844b8c4b7-pzj6t 5226m 7520Mi ds-cts-0 3281m 1703Mi ds-cts-1 3195m 1863Mi ds-cts-2 3617m 2022Mi ds-idrepo-0 8609m 1754Mi ds-idrepo-1 357m 996Mi ds-idrepo-2 344m 966Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 959Mi idm-b85f9d978-wnhxb 4m 992Mi lodemon-5cd7976d78-6npxn 13m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7956m 3214Mi 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG --- stdout --- 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG --- http response --- 19:47:25 INFO [http_cmd]: http status code OK 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- stderr --- 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 69m 0% 5427Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8743m 29% 4570Mi 4% gke-xlou-cdm-default-pool-1b0769c8-nhlf 5913m 19% 9022Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 5487m 18% 10358Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 3037m 10% 4833Mi 4% gke-xlou-cdm-default-pool-1b0769c8-wpwl 152m 0% 5021Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2247Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 26926m 90% 12278Mi 10% gke-xlou-cdm-ds-0f601751-hmm7 52m 0% 1479Mi 1% gke-xlou-cdm-ds-0f601751-j488 405m 1% 3429Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 3332m 11% 5722Mi 5% gke-xlou-cdm-ds-0f601751-nj6q 418m 1% 3986Mi 3% gke-xlou-cdm-ds-0f601751-nmxm 3628m 12% 4276Mi 3% gke-xlou-cdm-frontend-a8771548-i82n 8183m 51% 6794Mi 11% 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- status code --- 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:47:25 DEBUG 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG --- stderr --- 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:47:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:47:25 DEBUG { 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:47:25 DEBUG 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG --- http response --- 19:47:25 DEBUG { 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG { 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:47:25 DEBUG { 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG { 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "data": { 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "data": { 19:47:25 DEBUG { 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "data": { 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "data": { 19:47:25 DEBUG ----- output ----- 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "data": { 19:47:25 DEBUG "data": { 19:47:25 DEBUG "data": { 19:47:25 DEBUG "data": { 19:47:25 DEBUG "data": { 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "data": { 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG { 19:47:25 DEBUG "data": { 19:47:25 DEBUG "data": { 19:47:25 DEBUG "data": { 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "result": [ 19:47:25 DEBUG { 19:47:25 DEBUG "data": { 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG { 19:47:25 DEBUG "data": { 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "data": { 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "data": { 19:47:25 DEBUG "data": { 19:47:25 DEBUG "data": { 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "data": { 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "data": { 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "data": { 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "status": "success", 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "result": [ 19:47:25 DEBUG { 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "data": { 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "instance": "10.142.0.107:9100" 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG { 19:47:25 DEBUG "data": { 19:47:25 DEBUG "result": [ 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG "result": [ 19:47:25 DEBUG { 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "result": [ 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG "result": [ 19:47:25 DEBUG }, 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "resultType": "vector", 19:47:25 DEBUG { 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "metric": { 19:47:25 DEBUG { 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "instance": "10.142.0.107:9100" 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "metric": { 19:47:25 DEBUG { 19:47:25 DEBUG "value": [ 19:47:25 DEBUG { 19:47:25 DEBUG { 19:47:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:47:25 DEBUG "result": [ 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:47:25 DEBUG "pod": "ds-idrepo-0" 19:47:25 DEBUG "instance": "10.142.0.107:9100" 19:47:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:47:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "instance": "10.142.0.107:9100" 19:47:25 DEBUG }, 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "pod": "ds-cts-0" 19:47:25 DEBUG "pod": "ds-cts-0" 19:47:25 DEBUG { 19:47:25 DEBUG }, 19:47:25 DEBUG { 19:47:25 DEBUG "pod": "ds-idrepo-0" 19:47:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:47:25 DEBUG "pod": "ds-idrepo-0" 19:47:25 DEBUG "metric": { 19:47:25 DEBUG 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "metric": { 19:47:25 DEBUG }, 19:47:25 DEBUG { 19:47:25 DEBUG "pod": "ds-cts-0" 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG "pod": "ds-idrepo-0" 19:47:25 DEBUG }, 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "metric": { 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG "metric": { 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:47:25 DEBUG 19:47:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:47:25 DEBUG "instance": "10.142.0.107:9100" 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "metric": { 19:47:25 DEBUG }, 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:47:25 DEBUG "metric": { 19:47:25 DEBUG 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "value": [ 19:47:25 DEBUG }, 19:47:25 DEBUG 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG 19:47:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG }, 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG }, 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG }, 19:47:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG "value": [ 19:47:25 DEBUG [...] 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG }, 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG "value": [ 19:47:25 DEBUG }, 19:47:25 DEBUG [...] 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG [...] 19:47:25 DEBUG [...] 19:47:25 DEBUG [...] 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG [...] 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG [...] 19:47:25 DEBUG [...] 19:47:25 DEBUG [...] 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG [...] 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG 19:47:25 DEBUG "instance": "10.142.0.43:9100" 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG "instance": "10.142.0.43:9100" 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG }, 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG [...] 19:47:25 DEBUG }, 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG }, 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:47:25 DEBUG "pod": "ds-idrepo-2" 19:47:25 DEBUG "instance": "10.142.0.43:9100" 19:47:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:47:25 DEBUG 19:47:25 DEBUG }, 19:47:25 DEBUG "instance": "10.142.0.43:9100" 19:47:25 DEBUG 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "pod": "ds-cts-2" 19:47:25 DEBUG "pod": "ds-cts-2" 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "pod": "ds-idrepo-2" 19:47:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:47:25 DEBUG "pod": "ds-idrepo-2" 19:47:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG }, 19:47:25 DEBUG 19:47:25 DEBUG "pod": "ds-cts-2" 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG "value": [ 19:47:25 DEBUG }, 19:47:25 DEBUG "value": [ 19:47:25 DEBUG }, 19:47:25 DEBUG 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG 19:47:25 DEBUG 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:47:25 DEBUG "instance": "10.142.0.43:9100" 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG }, 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG "8.035473856624995" 19:47:25 DEBUG 19:47:25 DEBUG "3467407360" 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:47:25 DEBUG }, 19:47:25 DEBUG "27295815.8" 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "49376527360" 19:47:25 DEBUG }, 19:47:25 DEBUG }, 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:47:25 DEBUG "value": [ 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG "pod": "ds-cts-2" 19:47:25 DEBUG "0.07433333333333329" 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG "7.994058353764299" 19:47:25 DEBUG ] 19:47:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:47:25 DEBUG ] 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG 19:47:25 DEBUG "value": [ 19:47:25 DEBUG ] 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG ] 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "1302.2" 19:47:25 DEBUG }, 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG "0" 19:47:25 DEBUG "12.366666666666667" 19:47:25 DEBUG }, 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG }, 19:47:25 DEBUG } 19:47:25 DEBUG "0.000017660000000000004" 19:47:25 DEBUG "0" 19:47:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG } 19:47:25 DEBUG "65.1" 19:47:25 DEBUG "0" 19:47:25 DEBUG "14.233333333333333" 19:47:25 DEBUG "0" 19:47:25 DEBUG } 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG ] 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "0" 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG "value": [ 19:47:25 DEBUG "value": [ 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG "value": [ 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG }, 19:47:25 DEBUG "67731" 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG "35457012.74919669" 19:47:25 DEBUG "27294093.966666665" 19:47:25 DEBUG } 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG "value": [ 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG "1276.1666666666667" 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG "0" 19:47:25 DEBUG "0.11556666666666568" 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG "0.008" 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG 1713379645, 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG "0" 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG ] 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:47:25 DEBUG } 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG { 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "data": { 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "result": [ 19:48:25 DEBUG { 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:48:25 DEBUG }, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:48:25 DEBUG }, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "1333.9666666666667" 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713379705" 19:48:25 ERROR [http_cmd]: ERROR 19:48:25 ERROR --- status code --- 19:48:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:48:25 ERROR --- http response --- 19:48:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:48:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:48:25 INFO 19:48:25 INFO [loop_until]: kubectl --namespace=xlou top pods 19:48:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:48:25 INFO 19:48:25 INFO [loop_until]: kubectl --namespace=xlou top node 19:48:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "data": { 19:48:25 DEBUG "data": { 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "result": [ 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "metric": { 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:48:25 DEBUG "pod": "ds-cts-0" 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG }, 19:48:25 DEBUG }, 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG { 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG "data": { 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG "result": [ 19:48:25 DEBUG 19:48:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:48:25 DEBUG { 19:48:25 DEBUG "pod": "ds-cts-2" 19:48:25 DEBUG }, 19:48:25 DEBUG "metric": { 19:48:25 DEBUG }, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "instance": "10.142.0.107:9100" 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG }, 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "1333.4666666666667" 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "0" 19:48:25 DEBUG ] 19:48:25 DEBUG 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG [...] 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG 19:48:25 DEBUG } 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG "instance": "10.142.0.43:9100" 19:48:25 DEBUG }, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "0.022333333333333254" 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379705" 19:48:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379705" 19:48:25 INFO [loop_until]: OK (rc = 0) 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 DEBUG --- status code --- 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 INFO [loop_until]: OK (rc = 0) 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 DEBUG --- stdout --- 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG --- status code --- 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 DEBUG --- status code --- 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG --- stdout --- 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 8353m 8541Mi am-6844b8c4b7-n64np 5754m 5962Mi am-6844b8c4b7-pzj6t 5143m 7541Mi ds-cts-0 3044m 2675Mi ds-cts-1 2816m 2633Mi ds-cts-2 3482m 2848Mi ds-idrepo-0 8796m 2040Mi ds-idrepo-1 323m 1086Mi ds-idrepo-2 336m 1069Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 960Mi idm-b85f9d978-wnhxb 6m 992Mi lodemon-5cd7976d78-6npxn 12m 62Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7934m 5767Mi 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5440Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8616m 28% 4881Mi 4% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6081m 20% 8550Mi 7% gke-xlou-cdm-default-pool-1b0769c8-qkns 5547m 18% 10382Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 3069m 10% 5880Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 138m 0% 5013Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 56m 0% 2239Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 26435m 88% 12281Mi 11% gke-xlou-cdm-ds-0f601751-hmm7 51m 0% 1480Mi 1% gke-xlou-cdm-ds-0f601751-j488 381m 1% 3541Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 2841m 9% 6661Mi 5% gke-xlou-cdm-ds-0f601751-nj6q 407m 1% 4079Mi 3% gke-xlou-cdm-ds-0f601751-nmxm 3536m 11% 5350Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 8194m 51% 9123Mi 15% 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG --- stderr --- 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG --- stderr --- 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG --- http response --- 19:48:25 INFO [http_cmd]: http status code OK 19:48:25 DEBUG 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:48:25 DEBUG --- status code --- 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:48:25 DEBUG { 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:48:25 DEBUG { 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG --- http response --- 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG "data": { 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG { 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG { 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "data": { 19:48:25 DEBUG "data": { 19:48:25 DEBUG "data": { 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG { 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG "data": { 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "data": { 19:48:25 DEBUG "data": { 19:48:25 DEBUG { 19:48:25 DEBUG "data": { 19:48:25 DEBUG "data": { 19:48:25 DEBUG "data": { 19:48:25 DEBUG "data": { 19:48:25 DEBUG "data": { 19:48:25 DEBUG "data": { 19:48:25 DEBUG ----- output ----- 19:48:25 DEBUG "data": { 19:48:25 DEBUG "data": { 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "data": { 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG { 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "data": { 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG { 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG { 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "data": { 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "data": { 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "status": "success", 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "metric": { 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "data": { 19:48:25 DEBUG { 19:48:25 DEBUG "result": [ 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG "data": { 19:48:25 DEBUG { 19:48:25 DEBUG { 19:48:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "result": [ 19:48:25 DEBUG { 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "resultType": "vector", 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "metric": { 19:48:25 DEBUG }, 19:48:25 DEBUG "instance": "10.142.0.107:9100" 19:48:25 DEBUG "pod": "ds-idrepo-0" 19:48:25 DEBUG { 19:48:25 DEBUG }, 19:48:25 DEBUG { 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "pod": "ds-cts-0" 19:48:25 DEBUG { 19:48:25 DEBUG "instance": "10.142.0.107:9100" 19:48:25 DEBUG "pod": "ds-idrepo-0" 19:48:25 DEBUG { 19:48:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:48:25 DEBUG "pod": "ds-idrepo-0" 19:48:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:48:25 DEBUG "instance": "10.142.0.107:9100" 19:48:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:48:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:48:25 DEBUG "result": [ 19:48:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:48:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 19:48:25 DEBUG }, 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "metric": { 19:48:25 DEBUG }, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG }, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG }, 19:48:25 DEBUG }, 19:48:25 DEBUG }, 19:48:25 DEBUG }, 19:48:25 DEBUG }, 19:48:25 DEBUG }, 19:48:25 DEBUG { 19:48:25 DEBUG }, 19:48:25 DEBUG }, 19:48:25 DEBUG 19:48:25 DEBUG }, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:48:25 DEBUG 19:48:25 DEBUG "pod": "ds-idrepo-0" 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "pod": "ds-cts-0" 19:48:25 DEBUG }, 19:48:25 DEBUG 19:48:25 DEBUG "metric": { 19:48:25 DEBUG 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 19:48:25 DEBUG }, 19:48:25 DEBUG 19:48:25 DEBUG }, 19:48:25 DEBUG }, 19:48:25 DEBUG "metric": { 19:48:25 DEBUG "value": [ 19:48:25 DEBUG [...] 19:48:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG "instance": "10.142.0.107:9100" 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG }, 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG }, 19:48:25 DEBUG 19:48:25 DEBUG }, 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "pod": "ds-cts-2" 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG [...] 19:48:25 DEBUG [...] 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG }, 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "pod": "ds-idrepo-2" 19:48:25 DEBUG 19:48:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:48:25 DEBUG }, 19:48:25 DEBUG "instance": "10.142.0.43:9100" 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG }, 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG 19:48:25 DEBUG [...] 19:48:25 DEBUG "0" 19:48:25 DEBUG }, 19:48:25 DEBUG 19:48:25 DEBUG }, 19:48:25 DEBUG "pod": "ds-idrepo-2" 19:48:25 DEBUG "instance": "10.142.0.43:9100" 19:48:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:48:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:48:25 DEBUG 19:48:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:48:25 DEBUG }, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG }, 19:48:25 DEBUG "pod": "ds-idrepo-2" 19:48:25 DEBUG 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG "instance": "10.142.0.43:9100" 19:48:25 DEBUG }, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:48:25 DEBUG "value": [ 19:48:25 DEBUG }, 19:48:25 DEBUG }, 19:48:25 DEBUG }, 19:48:25 DEBUG }, 19:48:25 DEBUG 19:48:25 DEBUG }, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "6145028096" 19:48:25 DEBUG }, 19:48:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "pod": "ds-cts-2" 19:48:25 DEBUG "pod": "ds-cts-2" 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG ] 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG }, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG ] 19:48:25 DEBUG "value": [ 19:48:25 DEBUG }, 19:48:25 DEBUG "5.62896002414073" 19:48:25 DEBUG }, 19:48:25 DEBUG }, 19:48:25 DEBUG 19:48:25 DEBUG 19:48:25 DEBUG } 19:48:25 DEBUG "50606541824" 19:48:25 DEBUG "6.4" 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "0.008" 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "instance": "10.142.0.43:9100" 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "8.03732888939817" 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG } 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG ] 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "value": [ 19:48:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:48:25 DEBUG 19:48:25 DEBUG ] 19:48:25 DEBUG ] 19:48:25 DEBUG ] 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "0" 19:48:25 DEBUG ] 19:48:25 DEBUG "28933458.633333333" 19:48:25 DEBUG "0" 19:48:25 DEBUG "0" 19:48:25 DEBUG }, 19:48:25 DEBUG "42039581.08728232" 19:48:25 DEBUG ] 19:48:25 DEBUG "28933575.03333333" 19:48:25 DEBUG ] 19:48:25 DEBUG "8.266666666666666" 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG } 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "0" 19:48:25 DEBUG }, 19:48:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG "59.13333333333333" 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG ] 19:48:25 DEBUG ] 19:48:25 DEBUG "value": [ 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG "146973.5" 19:48:25 DEBUG ] 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG ] 19:48:25 DEBUG "value": [ 19:48:25 DEBUG }, 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG ] 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG "0.02993333333333297" 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG "0.000017660000000000004" 19:48:25 DEBUG } 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG "value": [ 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG ] 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG ] 19:48:25 DEBUG ] 19:48:25 DEBUG "1342.1" 19:48:25 DEBUG 1713379705, 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG "0" 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG ] 19:48:25 DEBUG ] 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:48:25 DEBUG } 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG "data": { 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "result": [ 19:49:25 DEBUG { 19:49:25 DEBUG "metric": { 19:49:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG "1261.2" 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713379765" 19:49:25 ERROR [http_cmd]: ERROR 19:49:25 ERROR --- status code --- 19:49:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:49:25 ERROR --- http response --- 19:49:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:49:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG "data": { 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "result": [ 19:49:25 DEBUG { 19:49:25 DEBUG "metric": { 19:49:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG "1260.8666666666666" 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG "data": { 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "result": [ 19:49:25 DEBUG { 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG "metric": { 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG "pod": "ds-cts-0" 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG }, 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:49:25 DEBUG 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG [...] 19:49:25 DEBUG { 19:49:25 DEBUG 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG 19:49:25 DEBUG "data": { 19:49:25 DEBUG 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "pod": "ds-cts-2" 19:49:25 DEBUG "result": [ 19:49:25 DEBUG }, 19:49:25 DEBUG { 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "metric": { 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG "instance": "10.142.0.107:9100" 19:49:25 DEBUG "0" 19:49:25 DEBUG }, 19:49:25 DEBUG ] 19:49:25 DEBUG "value": [ 19:49:25 DEBUG } 19:49:25 DEBUG 19:49:25 DEBUG ] 19:49:25 DEBUG 19:49:25 DEBUG } 19:49:25 DEBUG 19:49:25 DEBUG } 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "instance": "10.142.0.43:9100" 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG "0.022666666666666606" 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO 19:49:25 INFO 19:49:25 INFO [loop_until]: kubectl --namespace=xlou top pods 19:49:25 INFO [loop_until]: kubectl --namespace=xlou top node 19:49:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:49:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG "data": { 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "result": [ 19:49:25 DEBUG { 19:49:25 DEBUG "metric": { 19:49:25 DEBUG "pod": "ds-idrepo-0" 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "pod": "ds-idrepo-2" 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG "59" 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG "data": { 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "result": [ 19:49:25 DEBUG { 19:49:25 DEBUG "metric": { 19:49:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG "0" 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG "data": { 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "result": [ 19:49:25 DEBUG { 19:49:25 DEBUG "metric": { 19:49:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG "4532627.193772749" 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG --- http response --- 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713379765" 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG ----- output ----- 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG { 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379765" 19:49:25 DEBUG --- http response --- 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG "status": "success", 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG --- status code --- 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 INFO [loop_until]: OK (rc = 0) 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG "data": { 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG --- status code --- 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG --- stdout --- 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 INFO [http_cmd]: http status code OK 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 8491m 8537Mi am-6844b8c4b7-n64np 5719m 5963Mi am-6844b8c4b7-pzj6t 5027m 7547Mi ds-cts-0 2939m 3876Mi ds-cts-1 2612m 3717Mi ds-cts-2 3607m 4119Mi ds-idrepo-0 8506m 2257Mi ds-idrepo-1 305m 1161Mi ds-idrepo-2 306m 1141Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 960Mi idm-b85f9d978-wnhxb 6m 992Mi lodemon-5cd7976d78-6npxn 11m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7923m 6984Mi 19:49:25 DEBUG { 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG "result": [ 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG --- status code --- 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG --- stderr --- 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG { 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:49:25 INFO [loop_until]: OK (rc = 0) 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:49:25 DEBUG 19:49:25 DEBUG "data": { 19:49:25 DEBUG "metric": { 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG --- stdout --- 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG { 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG { 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "pod": "ds-idrepo-0" 19:49:25 DEBUG { 19:49:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5442Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8285m 27% 5105Mi 4% gke-xlou-cdm-default-pool-1b0769c8-nhlf 5845m 19% 8206Mi 7% gke-xlou-cdm-default-pool-1b0769c8-qkns 5510m 18% 10393Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 3013m 10% 6025Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 154m 0% 5012Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2250Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 25425m 85% 12286Mi 11% gke-xlou-cdm-ds-0f601751-hmm7 50m 0% 1480Mi 1% gke-xlou-cdm-ds-0f601751-j488 353m 1% 3610Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 2829m 9% 7676Mi 6% gke-xlou-cdm-ds-0f601751-nj6q 376m 1% 4174Mi 3% gke-xlou-cdm-ds-0f601751-nmxm 3474m 11% 5514Mi 4% gke-xlou-cdm-frontend-a8771548-i82n 8210m 51% 10409Mi 17% 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG { 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG { 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG --- http response --- 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG { 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:49:25 DEBUG { 19:49:25 DEBUG "result": [ 19:49:25 DEBUG }, 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG --- stderr --- 19:49:25 DEBUG { 19:49:25 DEBUG { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG { 19:49:25 DEBUG "data": { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG { 19:49:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:49:25 DEBUG "data": { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG { 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "data": { 19:49:25 DEBUG 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG "data": { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG "data": { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "data": { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG ----- output ----- 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "data": { 19:49:25 DEBUG { 19:49:25 DEBUG { 19:49:25 DEBUG "data": { 19:49:25 DEBUG "metric": { 19:49:25 DEBUG 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "data": { 19:49:25 DEBUG "data": { 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "data": { 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "data": { 19:49:25 DEBUG "result": [ 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "data": { 19:49:25 DEBUG { 19:49:25 DEBUG "result": [ 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "result": [ 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "pod": "ds-idrepo-0" 19:49:25 DEBUG 19:49:25 DEBUG "result": [ 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "result": [ 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "result": [ 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG { 19:49:25 DEBUG "result": [ 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG { 19:49:25 DEBUG "status": "success", 19:49:25 DEBUG { 19:49:25 DEBUG "data": { 19:49:25 DEBUG "result": [ 19:49:25 DEBUG }, 19:49:25 DEBUG 19:49:25 DEBUG { 19:49:25 DEBUG "result": [ 19:49:25 DEBUG "result": [ 19:49:25 DEBUG { 19:49:25 DEBUG "result": [ 19:49:25 DEBUG { 19:49:25 DEBUG "result": [ 19:49:25 DEBUG "metric": { 19:49:25 DEBUG { 19:49:25 DEBUG "result": [ 19:49:25 DEBUG "data": { 19:49:25 DEBUG "metric": { 19:49:25 DEBUG "data": { 19:49:25 DEBUG "metric": { 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG { 19:49:25 DEBUG "value": [ 19:49:25 DEBUG [...] 19:49:25 DEBUG "metric": { 19:49:25 DEBUG { 19:49:25 DEBUG { 19:49:25 DEBUG "metric": { 19:49:25 DEBUG { 19:49:25 DEBUG "metric": { 19:49:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:49:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:49:25 DEBUG }, 19:49:25 DEBUG { 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:49:25 DEBUG "resultType": "vector", 19:49:25 DEBUG "instance": "10.142.0.107:9100" 19:49:25 DEBUG "result": [ 19:49:25 DEBUG "metric": { 19:49:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:49:25 DEBUG 19:49:25 DEBUG "pod": "ds-cts-0" 19:49:25 DEBUG "metric": { 19:49:25 DEBUG "metric": { 19:49:25 DEBUG "instance": "10.142.0.107:9100" 19:49:25 DEBUG "metric": { 19:49:25 DEBUG { 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "metric": { 19:49:25 DEBUG "metric": { 19:49:25 DEBUG "result": [ 19:49:25 DEBUG }, 19:49:25 DEBUG "result": [ 19:49:25 DEBUG }, 19:49:25 DEBUG { 19:49:25 DEBUG 19:49:25 DEBUG }, 19:49:25 DEBUG 19:49:25 DEBUG }, 19:49:25 DEBUG "instance": "10.142.0.107:9100" 19:49:25 DEBUG "pod": "ds-idrepo-0" 19:49:25 DEBUG }, 19:49:25 DEBUG "pod": "ds-cts-0" 19:49:25 DEBUG "metric": { 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 19:49:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:49:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:49:25 DEBUG { 19:49:25 DEBUG "value": [ 19:49:25 DEBUG { 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "metric": { 19:49:25 DEBUG 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 19:49:25 DEBUG "value": [ 19:49:25 DEBUG }, 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 19:49:25 DEBUG }, 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG }, 19:49:25 DEBUG }, 19:49:25 DEBUG "metric": { 19:49:25 DEBUG 19:49:25 DEBUG "metric": { 19:49:25 DEBUG 19:49:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "pod": "ds-idrepo-2" 19:49:25 DEBUG 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:49:25 DEBUG 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:49:25 DEBUG 19:49:25 DEBUG "instance": "10.142.0.107:9100" 19:49:25 DEBUG 19:49:25 DEBUG }, 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG }, 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG }, 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG }, 19:49:25 DEBUG 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "value": [ 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG [...] 19:49:25 DEBUG }, 19:49:25 DEBUG 19:49:25 DEBUG "value": [ 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG [...] 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG [...] 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "value": [ 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "6.266666666666667" 19:49:25 DEBUG 19:49:25 DEBUG [...] 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG [...] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "pod": "ds-idrepo-2" 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:49:25 DEBUG 19:49:25 DEBUG ] 19:49:25 DEBUG 19:49:25 DEBUG "8.133333333333333" 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG [...] 19:49:25 DEBUG "instance": "10.142.0.43:9100" 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG }, 19:49:25 DEBUG 19:49:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG [...] 19:49:25 DEBUG [...] 19:49:25 DEBUG }, 19:49:25 DEBUG 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG }, 19:49:25 DEBUG "pod": "ds-cts-2" 19:49:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 19:49:25 DEBUG }, 19:49:25 DEBUG [...] 19:49:25 DEBUG "instance": "10.142.0.43:9100" 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG "pod": "ds-cts-2" 19:49:25 DEBUG "instance": "10.142.0.43:9100" 19:49:25 DEBUG "pod": "ds-cts-2" 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG }, 19:49:25 DEBUG }, 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG }, 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG }, 19:49:25 DEBUG }, 19:49:25 DEBUG }, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "265.43333333333334" 19:49:25 DEBUG }, 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG }, 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG 19:49:25 DEBUG "8.030993628435507" 19:49:25 DEBUG "value": [ 19:49:25 DEBUG ] 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG }, 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG ] 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "8.338295938592353" 19:49:25 DEBUG "value": [ 19:49:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:49:25 DEBUG "instance": "10.142.0.43:9100" 19:49:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:49:25 DEBUG ] 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG } 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG "51872574464" 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG "0.00006865333333333334" 19:49:25 DEBUG } 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG ] 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG }, 19:49:25 DEBUG }, 19:49:25 DEBUG }, 19:49:25 DEBUG } 19:49:25 DEBUG "0" 19:49:25 DEBUG } 19:49:25 DEBUG "0" 19:49:25 DEBUG "29300126.833333332" 19:49:25 DEBUG "0" 19:49:25 DEBUG ] 19:49:25 DEBUG "220794.5" 19:49:25 DEBUG "7422291968" 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG "0.008" 19:49:25 DEBUG } 19:49:25 DEBUG "29301654.333333336" 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG "value": [ 19:49:25 DEBUG ] 19:49:25 DEBUG ] 19:49:25 DEBUG ] 19:49:25 DEBUG ] 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG ] 19:49:25 DEBUG ] 19:49:25 DEBUG ] 19:49:25 DEBUG ] 19:49:25 DEBUG ] 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG "value": [ 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG "1238.8333333333333" 19:49:25 DEBUG ] 19:49:25 DEBUG 1713379765, 19:49:25 DEBUG "281785.4337093229" 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG ] 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG "0.03106666666666721" 19:49:25 DEBUG ] 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG ] 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:49:25 DEBUG } 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "data": { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "result": [ 19:50:25 DEBUG { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "pod": "am-6844b8c4b7-n64np" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG "1263.2" 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713379825" 19:50:25 ERROR [http_cmd]: ERROR 19:50:25 ERROR --- status code --- 19:50:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:50:25 ERROR --- http response --- 19:50:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:50:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379825" 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "data": { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "result": [ 19:50:25 DEBUG { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "pod": "am-6844b8c4b7-n64np" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG "1263.5333333333333" 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "data": { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "result": [ 19:50:25 DEBUG { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "pod": "ds-cts-0" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "pod": "ds-cts-2" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG "0" 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "data": { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "result": [ 19:50:25 DEBUG { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "instance": "10.142.0.107:9100" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "instance": "10.142.0.43:9100" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG "0.02466666666666669" 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "data": { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "result": [ 19:50:25 DEBUG { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "pod": "ds-idrepo-0" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "pod": "ds-idrepo-2" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG "0" 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "data": { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "result": [ 19:50:25 DEBUG { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "pod": "am-6844b8c4b7-n64np" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG "0" 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "data": { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "result": [ 19:50:25 DEBUG { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "pod": "ds-idrepo-1" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG "4429475.779930673" 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 INFO 19:50:25 INFO [loop_until]: kubectl --namespace=xlou top node 19:50:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379825" 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "data": { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "result": [ 19:50:25 DEBUG { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "instance": "10.142.0.107:9100" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "instance": "10.142.0.43:9100" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 INFO 19:50:25 DEBUG 1713379825, 19:50:25 INFO [loop_until]: kubectl --namespace=xlou top pods 19:50:25 DEBUG "53289479168" 19:50:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "data": { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "result": [ 19:50:25 DEBUG { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "instance": "10.142.0.107:9100" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "instance": "10.142.0.43:9100" 19:50:25 DEBUG }, 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379825" 19:50:25 DEBUG "value": [ 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG "32889842.1" 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "data": { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "result": [ 19:50:25 DEBUG { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "instance": "10.142.0.107:9100" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "instance": "10.142.0.43:9100" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG "0.034333333333333604" 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "data": { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "result": [ 19:50:25 DEBUG { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG "0.008" 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713379825" 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713379825" 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- http response --- 19:50:25 INFO [loop_until]: OK (rc = 0) 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG ----- output ----- 19:50:25 INFO [loop_until]: OK (rc = 0) 19:50:25 DEBUG --- status code --- 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:50:25 DEBUG ----- output ----- 19:50:25 INFO [http_cmd]: http status code OK 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 19:50:25 DEBUG --- stdout --- 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:50:25 DEBUG { 19:50:25 DEBUG --- stdout --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG { 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG { 19:50:25 DEBUG --- status code --- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-6844b8c4b7-6m7z2 9921m 8594Mi am-6844b8c4b7-n64np 7115m 4917Mi am-7996cfcc7c-8j9vh 2741m 669Mi ds-cts-0 5963m 4411Mi ds-cts-1 5764m 4429Mi ds-cts-2 6955m 4471Mi ds-idrepo-0 9412m 2420Mi ds-idrepo-1 301m 1221Mi ds-idrepo-2 327m 1324Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 959Mi idm-b85f9d978-wnhxb 4m 992Mi lodemon-5cd7976d78-6npxn 17m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8178m 7882Mi 19:50:25 DEBUG { 19:50:25 DEBUG { 19:50:25 DEBUG { 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 5434Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9279m 31% 5244Mi 4% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7447m 24% 7508Mi 6% gke-xlou-cdm-default-pool-1b0769c8-qkns 6706m 22% 10402Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 5829m 19% 6527Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 135m 0% 5012Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2743m 9% 3552Mi 3% gke-xlou-cdm-ds-0f601751-cf7b 27472m 92% 12392Mi 11% gke-xlou-cdm-ds-0f601751-hmm7 49m 0% 1480Mi 1% gke-xlou-cdm-ds-0f601751-j488 375m 1% 3772Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 5749m 19% 8345Mi 7% gke-xlou-cdm-ds-0f601751-nj6q 357m 1% 4232Mi 3% gke-xlou-cdm-ds-0f601751-nmxm 7101m 23% 6034Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8201m 51% 11033Mi 18% 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG { 19:50:25 DEBUG { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG { 19:50:25 DEBUG --- stderr --- 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG { 19:50:25 DEBUG "data": { 19:50:25 DEBUG --- stderr --- 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:50:25 DEBUG "data": { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "data": { 19:50:25 DEBUG --- http response --- 19:50:25 DEBUG { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG 19:50:25 DEBUG "data": { 19:50:25 DEBUG "data": { 19:50:25 DEBUG "data": { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "data": { 19:50:25 DEBUG "data": { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:50:25 DEBUG "data": { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "data": { 19:50:25 DEBUG "result": [ 19:50:25 DEBUG { 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG "result": [ 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "result": [ 19:50:25 DEBUG "result": [ 19:50:25 DEBUG "data": { 19:50:25 DEBUG ----- output ----- 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "result": [ 19:50:25 DEBUG "result": [ 19:50:25 DEBUG "result": [ 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG { 19:50:25 DEBUG { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG { 19:50:25 DEBUG { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG { 19:50:25 DEBUG "result": [ 19:50:25 DEBUG { 19:50:25 DEBUG { 19:50:25 DEBUG { 19:50:25 DEBUG "result": [ 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "data": { 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "result": [ 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "result": [ 19:50:25 DEBUG "status": "success", 19:50:25 DEBUG { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG { 19:50:25 DEBUG "pod": "ds-idrepo-0" 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "data": { 19:50:25 DEBUG "pod": "ds-idrepo-1" 19:50:25 DEBUG { 19:50:25 DEBUG "pod": "ds-idrepo-1" 19:50:25 DEBUG "pod": "ds-idrepo-0" 19:50:25 DEBUG { 19:50:25 DEBUG "data": { 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:50:25 DEBUG "pod": "am-6844b8c4b7-n64np" 19:50:25 DEBUG "pod": "ds-cts-0" 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG "result": [ 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG }, 19:50:25 DEBUG "metric": { 19:50:25 DEBUG }, 19:50:25 DEBUG }, 19:50:25 DEBUG "metric": { 19:50:25 DEBUG "resultType": "vector", 19:50:25 DEBUG "pod": "am-6844b8c4b7-pzj6t" 19:50:25 DEBUG }, 19:50:25 DEBUG }, 19:50:25 DEBUG }, 19:50:25 DEBUG "metric": { 19:50:25 DEBUG 19:50:25 DEBUG { 19:50:25 DEBUG "result": [ 19:50:25 DEBUG "value": [ 19:50:25 DEBUG "instance": "10.142.0.107:9100" 19:50:25 DEBUG "value": [ 19:50:25 DEBUG "value": [ 19:50:25 DEBUG "pod": "am-6844b8c4b7-n64np" 19:50:25 DEBUG "result": [ 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG "pod": "ds-cts-0" 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG { 19:50:25 DEBUG 19:50:25 DEBUG }, 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG }, 19:50:25 DEBUG { 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG }, 19:50:25 DEBUG "metric": { 19:50:25 DEBUG [...] 19:50:25 DEBUG "metric": { 19:50:25 DEBUG 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "value": [ 19:50:25 DEBUG "metric": { 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "value": [ 19:50:25 DEBUG "pod": "ds-idrepo-1" 19:50:25 DEBUG 19:50:25 DEBUG "pod": "ds-idrepo-0" 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG }, 19:50:25 DEBUG 19:50:25 DEBUG }, 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG }, 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "value": [ 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG "pod": "ds-cts-2" 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "pod": "ds-idrepo-2" 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:50:25 DEBUG 19:50:25 DEBUG }, 19:50:25 DEBUG [...] 19:50:25 DEBUG 19:50:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:50:25 DEBUG 19:50:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:50:25 DEBUG }, 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:50:25 DEBUG 19:50:25 DEBUG }, 19:50:25 DEBUG 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG }, 19:50:25 DEBUG 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG [...] 19:50:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:50:25 DEBUG }, 19:50:25 DEBUG }, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "value": [ 19:50:25 DEBUG "instance": "10.142.0.43:9100" 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG "pod": "ds-idrepo-2" 19:50:25 DEBUG "value": [ 19:50:25 DEBUG "value": [ 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG "pod": "ds-cts-2" 19:50:25 DEBUG "0" 19:50:25 DEBUG 19:50:25 DEBUG 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG }, 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG "0" 19:50:25 DEBUG "pod": "am-6844b8c4b7-6m7z2" 19:50:25 DEBUG 19:50:25 DEBUG }, 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG "1345.4666666666667" 19:50:25 DEBUG }, 19:50:25 DEBUG ] 19:50:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:50:25 DEBUG 19:50:25 DEBUG "0" 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG 19:50:25 DEBUG "value": [ 19:50:25 DEBUG "232197" 19:50:25 DEBUG "8.02932068717943" 19:50:25 DEBUG ] 19:50:25 DEBUG "value": [ 19:50:25 DEBUG } 19:50:25 DEBUG }, 19:50:25 DEBUG "pod": "ds-cts-2" 19:50:25 DEBUG }, 19:50:25 DEBUG "0" 19:50:25 DEBUG ] 19:50:25 DEBUG }, 19:50:25 DEBUG } 19:50:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG ] 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG ] 19:50:25 DEBUG "value": [ 19:50:25 DEBUG "value": [ 19:50:25 DEBUG "value": [ 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG "value": [ 19:50:25 DEBUG }, 19:50:25 DEBUG "0" 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG "0" 19:50:25 DEBUG } 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG "value": [ 19:50:25 DEBUG ] 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG "0" 19:50:25 DEBUG "32890087.166666664" 19:50:25 DEBUG "0" 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG 1713379825, 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG ] 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG "1235.6666666666667" 19:50:25 DEBUG "8364814336" 19:50:25 DEBUG ] 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG ] 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG ] 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG } 19:50:25 DEBUG } 19:51:06 WARNING No values available for selected timeframe. 19:51:06 WARNING File /tmp/lodemon_data-20240417_194420/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [17/Apr/2024 19:51:06] "GET /monitoring/average?start_time=20240417_184727&stop_time=20240417_184906 HTTP/1.1" 200 - 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "data": { 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "result": [ 19:51:25 DEBUG { 19:51:25 DEBUG "metric": { 19:51:25 DEBUG "pod": "am-7996cfcc7c-8j9vh" 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG "0" 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713379885" 19:51:25 ERROR [http_cmd]: ERROR 19:51:25 ERROR --- status code --- 19:51:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:51:25 ERROR --- http response --- 19:51:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:51:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379885" 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG { 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "data": { 19:51:25 DEBUG "status": "success", 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG "data": { 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "data": { 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "result": [ 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG "result": [ 19:51:25 DEBUG { 19:51:25 DEBUG "result": [ 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG { 19:51:25 DEBUG "metric": { 19:51:25 DEBUG { 19:51:25 DEBUG "metric": { 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:51:25 DEBUG "pod": "am-7996cfcc7c-8j9vh" 19:51:25 DEBUG "metric": { 19:51:25 DEBUG "pod": "ds-idrepo-0" 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG }, 19:51:25 DEBUG "pod": "ds-cts-0" 19:51:25 DEBUG }, 19:51:25 DEBUG { 19:51:25 DEBUG "value": [ 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG "data": { 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "result": [ 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG { 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG "metric": { 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "instance": "10.142.0.107:9100" 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG }, 19:51:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:51:25 DEBUG 19:51:25 DEBUG "pod": "ds-idrepo-2" 19:51:25 DEBUG "value": [ 19:51:25 DEBUG }, 19:51:25 DEBUG "pod": "ds-cts-2" 19:51:25 DEBUG }, 19:51:25 DEBUG 19:51:25 DEBUG "value": [ 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG 19:51:25 DEBUG "0" 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG "0" 19:51:25 DEBUG [...] 19:51:25 DEBUG ] 19:51:25 DEBUG "0" 19:51:25 DEBUG ] 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 DEBUG 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG "instance": "10.142.0.43:9100" 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG }, 19:51:25 DEBUG } 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG "0.010000000000000023" 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "data": { 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "result": [ 19:51:25 DEBUG { 19:51:25 DEBUG "metric": { 19:51:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "pod": "ds-cts-0" 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG "10423032.86565443" 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "data": { 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "result": [ 19:51:25 DEBUG { 19:51:25 DEBUG "metric": { 19:51:25 DEBUG "pod": "am-7996cfcc7c-8j9vh" 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG "0" 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379885" 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "data": { 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "result": [ 19:51:25 DEBUG { 19:51:25 DEBUG "metric": { 19:51:25 DEBUG "instance": "10.142.0.107:9100" 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "instance": "10.142.0.43:9100" 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG "0.014300000000000306" 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379885" 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379885" 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "data": { 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "result": [ 19:51:25 DEBUG { 19:51:25 DEBUG "metric": { 19:51:25 DEBUG "instance": "10.142.0.107:9100" 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "instance": "10.142.0.43:9100" 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG "54263823360" 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "data": { 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "result": [ 19:51:25 DEBUG { 19:51:25 DEBUG "metric": { 19:51:25 DEBUG "instance": "10.142.0.107:9100" 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "instance": "10.142.0.43:9100" 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG "7166881.199999999" 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "data": { 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "result": [ 19:51:25 DEBUG { 19:51:25 DEBUG "metric": { 19:51:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG "0" 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379885" 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [loop_until]: kubectl --namespace=xlou top pods 19:51:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713379885" 19:51:25 INFO 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [loop_until]: kubectl --namespace=xlou top node 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG ----- output ----- 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG --- status code --- 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG { 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG --- status code --- 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG --- status code --- 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG "data": { 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:51:25 DEBUG "result": [ 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:51:25 DEBUG { 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:51:25 DEBUG { 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG { 19:51:25 DEBUG { 19:51:25 DEBUG { 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG { 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG { 19:51:25 DEBUG "metric": { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG { 19:51:25 DEBUG "data": { 19:51:25 DEBUG { 19:51:25 DEBUG "data": { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:51:25 DEBUG "data": { 19:51:25 DEBUG "data": { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "data": { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "data": { 19:51:25 DEBUG }, 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "data": { 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "data": { 19:51:25 DEBUG "result": [ 19:51:25 DEBUG "data": { 19:51:25 DEBUG "result": [ 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "value": [ 19:51:25 DEBUG "result": [ 19:51:25 DEBUG "result": [ 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "result": [ 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG { 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG { 19:51:25 DEBUG "result": [ 19:51:25 DEBUG 19:51:25 DEBUG { 19:51:25 DEBUG { 19:51:25 DEBUG "result": [ 19:51:25 DEBUG { 19:51:25 DEBUG "result": [ 19:51:25 DEBUG "metric": { 19:51:25 DEBUG "result": [ 19:51:25 DEBUG "metric": { 19:51:25 DEBUG { 19:51:25 DEBUG 19:51:25 DEBUG "metric": { 19:51:25 DEBUG "metric": { 19:51:25 DEBUG { 19:51:25 DEBUG "metric": { 19:51:25 DEBUG { 19:51:25 DEBUG "pod": "ds-cts-0" 19:51:25 DEBUG { 19:51:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:51:25 DEBUG "metric": { 19:51:25 DEBUG 19:51:25 DEBUG "pod": "ds-idrepo-0" 19:51:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:51:25 DEBUG "metric": { 19:51:25 DEBUG "instance": "10.142.0.107:9100" 19:51:25 DEBUG "metric": { 19:51:25 DEBUG }, 19:51:25 DEBUG "metric": { 19:51:25 DEBUG }, 19:51:25 DEBUG "pod": "am-7996cfcc7c-8j9vh" 19:51:25 DEBUG [...] 19:51:25 DEBUG }, 19:51:25 DEBUG }, 19:51:25 DEBUG "pod": "ds-idrepo-0" 19:51:25 DEBUG }, 19:51:25 DEBUG "pod": "ds-cts-0" 19:51:25 DEBUG "value": [ 19:51:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:51:25 DEBUG "value": [ 19:51:25 DEBUG }, 19:51:25 DEBUG 19:51:25 DEBUG "value": [ 19:51:25 DEBUG "value": [ 19:51:25 DEBUG }, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG }, 19:51:25 DEBUG 19:51:25 DEBUG }, 19:51:25 DEBUG 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG }, 19:51:25 DEBUG [...] 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG [...] 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "0" 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "instance": "10.142.0.43:9100" 19:51:25 DEBUG 19:51:25 DEBUG "pod": "ds-cts-2" 19:51:25 DEBUG 19:51:25 DEBUG "pod": "ds-cts-0" 19:51:25 DEBUG 19:51:25 DEBUG ] 19:51:25 DEBUG "pod": "ds-idrepo-2" 19:51:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:51:25 DEBUG 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379885" 19:51:25 DEBUG }, 19:51:25 DEBUG 19:51:25 DEBUG }, 19:51:25 DEBUG 19:51:25 DEBUG }, 19:51:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:51:25 DEBUG } 19:51:25 DEBUG }, 19:51:25 DEBUG }, 19:51:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713379885" 19:51:25 DEBUG "pod": "ds-idrepo-2" 19:51:25 DEBUG "value": [ 19:51:25 DEBUG "pod": "ds-cts-2" 19:51:25 DEBUG "value": [ 19:51:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:51:25 DEBUG "value": [ 19:51:25 DEBUG }, 19:51:25 DEBUG ] 19:51:25 DEBUG "value": [ 19:51:25 DEBUG "value": [ 19:51:25 DEBUG }, 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG }, 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG }, 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG } 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG "value": [ 19:51:25 DEBUG "7169508.800000001" 19:51:25 DEBUG "value": [ 19:51:25 DEBUG "0" 19:51:25 DEBUG "value": [ 19:51:25 DEBUG "0" 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG } 19:51:25 DEBUG "0" 19:51:25 DEBUG "7.261300343472388" 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG ] 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG ] 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG ] 19:51:25 DEBUG "0" 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG ] 19:51:25 DEBUG ] 19:51:25 DEBUG "0" 19:51:25 DEBUG } 19:51:25 DEBUG "0" 19:51:25 DEBUG } 19:51:25 DEBUG "0" 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG ] 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG ] 19:51:25 DEBUG ] 19:51:25 DEBUG ] 19:51:25 DEBUG ] 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG ] 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 INFO [loop_until]: OK (rc = 0) 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "pod": "am-6844b8c4b7-6m7z2" }, "value": [ 1713379885, "1325.5333333333333" ] } ] } } 19:51:25 DEBUG } 19:51:25 DEBUG ----- output ----- 19:51:25 INFO [http_cmd]: http status code OK 19:51:25 DEBUG } 19:51:25 DEBUG --- stdout --- 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG { 19:51:25 DEBUG --- status code --- 19:51:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-7996cfcc7c-8h7fs 4233m 2685Mi am-7996cfcc7c-8j9vh 4527m 4593Mi am-7996cfcc7c-989c6 2547m 674Mi ds-cts-0 4270m 4430Mi ds-cts-1 3215m 4459Mi ds-cts-2 5601m 4551Mi ds-idrepo-0 8567m 2601Mi ds-idrepo-1 278m 1300Mi ds-idrepo-2 227m 1349Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 8m 959Mi idm-b85f9d978-wnhxb 8m 992Mi lodemon-5cd7976d78-6npxn 12m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7403m 15Mi 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:51:25 DEBUG --- stderr --- 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG "data": { 19:51:25 DEBUG --- http response --- 19:51:25 DEBUG 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (44 lines): 19:51:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:51:25 DEBUG "result": [ 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG ----- output ----- 19:51:25 DEBUG { 19:51:25 DEBUG { 19:51:25 DEBUG { 19:51:25 DEBUG "metric": { 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "status": "success", 19:51:25 DEBUG "pod": "ds-idrepo-0" 19:51:25 DEBUG "data": { 19:51:25 DEBUG "data": { 19:51:25 DEBUG }, 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "resultType": "vector", 19:51:25 DEBUG "value": [ 19:51:25 DEBUG "result": [ 19:51:25 DEBUG "result": [ 19:51:25 DEBUG 19:51:25 DEBUG { 19:51:25 DEBUG { 19:51:25 DEBUG 19:51:25 DEBUG "metric": { 19:51:25 DEBUG "metric": { 19:51:25 DEBUG 19:51:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:51:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:51:25 DEBUG [...] 19:51:25 DEBUG }, 19:51:25 DEBUG }, 19:51:25 DEBUG 19:51:25 DEBUG "value": [ 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "pod": "ds-cts-2" 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG }, 19:51:25 DEBUG [...] 19:51:25 DEBUG [...] 19:51:25 DEBUG "value": [ 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG "0" 19:51:25 DEBUG 19:51:25 DEBUG 19:51:25 DEBUG ] 19:51:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:51:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:51:25 DEBUG } 19:51:25 DEBUG }, 19:51:25 DEBUG }, 19:51:25 DEBUG ] 19:51:25 DEBUG "value": [ 19:51:25 DEBUG "value": [ 19:51:25 DEBUG } 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG 1713379885, 19:51:25 DEBUG } 19:51:25 DEBUG "1418" 19:51:25 DEBUG "115396608" 19:51:25 DEBUG ] 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG ] 19:51:25 DEBUG ] 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 DEBUG } 19:51:25 INFO [loop_until]: OK (rc = 0) 19:51:25 DEBUG --- stdout --- 19:51:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 70m 0% 5431Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 6634m 22% 5436Mi 4% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6330m 21% 2577Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 708m 2% 4013Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 3561m 11% 6556Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 172m 0% 5003Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 3267m 10% 7107Mi 6% gke-xlou-cdm-ds-0f601751-cf7b 16934m 56% 11642Mi 10% gke-xlou-cdm-ds-0f601751-hmm7 3817m 12% 4434Mi 3% gke-xlou-cdm-ds-0f601751-j488 261m 0% 3831Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 3683m 12% 7877Mi 7% gke-xlou-cdm-ds-0f601751-nj6q 261m 0% 4306Mi 3% gke-xlou-cdm-ds-0f601751-nmxm 4514m 15% 6133Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 5638m 35% 3834Mi 6% 19:51:25 DEBUG --- stderr --- 19:51:25 DEBUG 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "1431.9393294444444" 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713379945" 19:52:25 ERROR [http_cmd]: ERROR 19:52:25 ERROR --- status code --- 19:52:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:52:25 ERROR --- http response --- 19:52:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:52:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379945" 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "pod": "ds-cts-0" 19:52:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:52:25 DEBUG }, 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "pod": "ds-cts-2" 19:52:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:52:25 DEBUG }, 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "0" 19:52:25 DEBUG "1431.7404811111112" 19:52:25 DEBUG ] 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "pod": "ds-idrepo-0" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "pod": "ds-idrepo-2" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "0" 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "0" 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "instance": "10.142.0.107:9100" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "instance": "10.142.0.43:9100" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "0.028000000000000077" 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "0" 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379945" 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379945" 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "instance": "10.142.0.107:9100" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "instance": "10.142.0.43:9100" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "0.03726666666666673" 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379945" 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "instance": "10.142.0.107:9100" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "instance": "10.142.0.43:9100" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "55168453632" 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "instance": "10.142.0.107:9100" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "instance": "10.142.0.43:9100" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "41430971.766666666" 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "0" 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "pod": "ds-cts-0" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "pod": "ds-cts-2" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "0" 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713379945" 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "instance": "10.142.0.107:9100" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "instance": "10.142.0.43:9100" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "41431581.36666667" 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713379945" 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [loop_until]: kubectl --namespace=xlou top pods 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713379945" 19:52:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713379945" 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- status code --- 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG --- status code --- 19:52:25 INFO [http_cmd]: http status code OK 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG --- status code --- 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:52:25 DEBUG { 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:52:25 DEBUG --- http response --- 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG { 19:52:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:52:25 DEBUG { 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG { 19:52:25 DEBUG "data": { 19:52:25 DEBUG { 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG ----- output ----- 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "data": { 19:52:25 DEBUG { 19:52:25 DEBUG "data": { 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG { 19:52:25 DEBUG "data": { 19:52:25 DEBUG "result": [ 19:52:25 DEBUG "data": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "status": "success", 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG "result": [ 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG "result": [ 19:52:25 DEBUG "data": { 19:52:25 DEBUG "result": [ 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG { 19:52:25 DEBUG "pod": "ds-idrepo-0" 19:52:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:52:25 DEBUG "result": [ 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "result": [ 19:52:25 DEBUG { 19:52:25 DEBUG "resultType": "vector", 19:52:25 DEBUG { 19:52:25 DEBUG { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:52:25 DEBUG }, 19:52:25 DEBUG }, 19:52:25 DEBUG { 19:52:25 DEBUG "pod": "ds-cts-0" 19:52:25 DEBUG { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "result": [ 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "metric": { 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG "value": [ 19:52:25 DEBUG "metric": { 19:52:25 DEBUG }, 19:52:25 DEBUG "metric": { 19:52:25 DEBUG "pod": "ds-idrepo-0" 19:52:25 DEBUG { 19:52:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:52:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:52:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "pod": "ds-idrepo-0" 19:52:25 INFO 19:52:25 DEBUG "value": [ 19:52:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:52:25 DEBUG }, 19:52:25 DEBUG "metric": { 19:52:25 DEBUG }, 19:52:25 DEBUG }, 19:52:25 DEBUG }, 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG }, 19:52:25 INFO [loop_until]: kubectl --namespace=xlou top node 19:52:25 DEBUG 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:52:25 DEBUG "value": [ 19:52:25 DEBUG "value": [ 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "value": [ 19:52:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:52:25 DEBUG 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG }, 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "pod": "ds-idrepo-2" 19:52:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:52:25 DEBUG }, 19:52:25 DEBUG }, 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG [...] 19:52:25 DEBUG "pod": "ds-idrepo-2" 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG "value": [ 19:52:25 DEBUG "pod": "ds-cts-2" 19:52:25 DEBUG "pod": "ds-cts-2" 19:52:25 DEBUG 19:52:25 DEBUG 19:52:25 DEBUG }, 19:52:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:52:25 DEBUG 19:52:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG }, 19:52:25 DEBUG }, 19:52:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:52:25 DEBUG 19:52:25 DEBUG "value": [ 19:52:25 DEBUG }, 19:52:25 DEBUG }, 19:52:25 DEBUG }, 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "0" 19:52:25 DEBUG "1.966231754359247" 19:52:25 DEBUG "value": [ 19:52:25 DEBUG "value": [ 19:52:25 DEBUG }, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "39711" 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG "value": [ 19:52:25 DEBUG "3.8916444852660126" 19:52:25 DEBUG ] 19:52:25 DEBUG ] 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG 19:52:25 DEBUG "value": [ 19:52:25 DEBUG "value": [ 19:52:25 DEBUG ] 19:52:25 DEBUG "0" 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG "0" 19:52:25 DEBUG "0" 19:52:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG "0" 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG ] 19:52:25 DEBUG ] 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG "1809.0666666666666" 19:52:25 DEBUG "458752.96195396903" 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG }, 19:52:25 DEBUG ] 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG "value": [ 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG 1713379945, 19:52:25 DEBUG ] 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG "653537280" 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 DEBUG ] 19:52:25 DEBUG } 19:52:25 DEBUG } 19:52:25 INFO [loop_until]: OK (rc = 0) 19:52:25 DEBUG --- stdout --- 19:52:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-7996cfcc7c-8h7fs 4861m 4233Mi am-7996cfcc7c-8j9vh 3160m 4782Mi am-7996cfcc7c-989c6 5206m 3731Mi ds-cts-0 944m 4438Mi ds-cts-1 1195m 4491Mi ds-cts-2 1053m 4558Mi ds-idrepo-0 4110m 2594Mi ds-idrepo-1 264m 1326Mi ds-idrepo-2 176m 1400Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 960Mi idm-b85f9d978-wnhxb 9m 992Mi lodemon-5cd7976d78-6npxn 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 3874m 528Mi 19:52:25 DEBUG --- stderr --- 19:52:25 DEBUG 19:52:26 INFO [loop_until]: OK (rc = 0) 19:52:26 DEBUG --- stdout --- 19:52:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 73m 0% 5438Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4407m 14% 5448Mi 4% gke-xlou-cdm-default-pool-1b0769c8-nhlf 51m 0% 2569Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 6249m 20% 6662Mi 5% gke-xlou-cdm-default-pool-1b0769c8-w15q 2242m 7% 6559Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 163m 0% 5029Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 3804m 12% 7289Mi 6% gke-xlou-cdm-ds-0f601751-cf7b 10502m 35% 2935Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 7081m 23% 6108Mi 5% gke-xlou-cdm-ds-0f601751-j488 262m 0% 3879Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 1894m 6% 7898Mi 7% gke-xlou-cdm-ds-0f601751-nj6q 393m 1% 4330Mi 3% gke-xlou-cdm-ds-0f601751-nmxm 1710m 5% 6130Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 4009m 25% 4305Mi 7% 19:52:26 DEBUG --- stderr --- 19:52:26 DEBUG 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380005" 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380005" 19:53:25 INFO [http_cmd]: http status code OK 19:53:25 INFO [http_cmd]: http status code OK 19:53:25 DEBUG --- status code --- 19:53:25 DEBUG --- status code --- 19:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:25 DEBUG --- http response --- 19:53:25 DEBUG --- http response --- 19:53:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:53:25 DEBUG ----- output ----- 19:53:25 DEBUG { 19:53:25 DEBUG "status": "success", 19:53:25 DEBUG "data": { 19:53:25 DEBUG "resultType": "vector", 19:53:25 DEBUG "result": [ 19:53:25 DEBUG { 19:53:25 DEBUG "metric": { 19:53:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG [...] 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 1713380005, 19:53:25 DEBUG "1115.5666666666666" 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG } 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713380005" 19:53:25 ERROR [http_cmd]: ERROR 19:53:25 ERROR --- status code --- 19:53:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:53:25 ERROR --- http response --- 19:53:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:53:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380005" 19:53:25 INFO [http_cmd]: http status code OK 19:53:25 DEBUG --- status code --- 19:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:25 DEBUG --- http response --- 19:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:53:25 DEBUG ----- output ----- 19:53:25 DEBUG { 19:53:25 DEBUG "status": "success", 19:53:25 DEBUG "data": { 19:53:25 DEBUG "resultType": "vector", 19:53:25 DEBUG "result": [ 19:53:25 DEBUG { 19:53:25 DEBUG "metric": { 19:53:25 DEBUG "pod": "ds-cts-0" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG [...] 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG "pod": "ds-cts-2" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 1713380005, 19:53:25 DEBUG "0" 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG } 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380005" 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380005" 19:53:25 INFO [http_cmd]: http status code OK 19:53:25 DEBUG --- status code --- 19:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:25 DEBUG --- http response --- 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380005" 19:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380005" 19:53:25 DEBUG ----- output ----- 19:53:25 INFO [http_cmd]: http status code OK 19:53:25 DEBUG { 19:53:25 DEBUG --- status code --- 19:53:25 DEBUG "status": "success", 19:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:25 DEBUG "data": { 19:53:25 DEBUG --- http response --- 19:53:25 DEBUG "resultType": "vector", 19:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:53:25 DEBUG "result": [ 19:53:25 DEBUG ----- output ----- 19:53:25 DEBUG { 19:53:25 DEBUG { 19:53:25 DEBUG "metric": { 19:53:25 DEBUG "status": "success", 19:53:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:53:25 DEBUG "data": { 19:53:25 DEBUG }, 19:53:25 DEBUG "resultType": "vector", 19:53:25 DEBUG "value": [ 19:53:25 DEBUG "result": [ 19:53:25 DEBUG 19:53:25 DEBUG { 19:53:25 DEBUG 19:53:25 DEBUG "metric": { 19:53:25 DEBUG 19:53:25 DEBUG "pod": "ds-idrepo-0" 19:53:25 DEBUG [...] 19:53:25 DEBUG }, 19:53:25 DEBUG 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:53:25 DEBUG 19:53:25 DEBUG }, 19:53:25 DEBUG [...] 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 19:53:25 DEBUG 1713380005, 19:53:25 DEBUG 19:53:25 DEBUG "1115.3" 19:53:25 DEBUG 19:53:25 DEBUG ] 19:53:25 DEBUG "pod": "ds-idrepo-2" 19:53:25 DEBUG } 19:53:25 DEBUG }, 19:53:25 DEBUG ] 19:53:25 DEBUG "value": [ 19:53:25 DEBUG } 19:53:25 DEBUG 1713380005, 19:53:25 DEBUG } 19:53:25 DEBUG "0" 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG } 19:53:25 INFO [http_cmd]: http status code OK 19:53:25 DEBUG --- status code --- 19:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:25 DEBUG --- http response --- 19:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:53:25 DEBUG ----- output ----- 19:53:25 DEBUG { 19:53:25 DEBUG "status": "success", 19:53:25 DEBUG "data": { 19:53:25 DEBUG "resultType": "vector", 19:53:25 DEBUG "result": [ 19:53:25 DEBUG { 19:53:25 DEBUG "metric": { 19:53:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG [...] 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 1713380005, 19:53:25 DEBUG "0" 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG } 19:53:25 INFO [http_cmd]: http status code OK 19:53:25 DEBUG --- status code --- 19:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:25 DEBUG --- http response --- 19:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:53:25 DEBUG ----- output ----- 19:53:25 DEBUG { 19:53:25 DEBUG "status": "success", 19:53:25 DEBUG "data": { 19:53:25 DEBUG "resultType": "vector", 19:53:25 DEBUG "result": [ 19:53:25 DEBUG { 19:53:25 DEBUG "metric": { 19:53:25 DEBUG "instance": "10.142.0.107:9100" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG [...] 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG "instance": "10.142.0.43:9100" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 1713380005, 19:53:25 DEBUG "0.021666666666666723" 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG } 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380005" 19:53:25 INFO [http_cmd]: http status code OK 19:53:25 DEBUG --- status code --- 19:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:25 DEBUG --- http response --- 19:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:53:25 DEBUG ----- output ----- 19:53:25 DEBUG { 19:53:25 DEBUG "status": "success", 19:53:25 DEBUG "data": { 19:53:25 DEBUG "resultType": "vector", 19:53:25 DEBUG "result": [ 19:53:25 DEBUG { 19:53:25 DEBUG "metric": { 19:53:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG [...] 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 1713380005, 19:53:25 DEBUG "0" 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG } 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380005" 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380005" 19:53:25 INFO [http_cmd]: http status code OK 19:53:25 INFO [http_cmd]: http status code OK 19:53:25 DEBUG --- status code --- 19:53:25 DEBUG --- status code --- 19:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:25 DEBUG --- http response --- 19:53:25 DEBUG --- http response --- 19:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:53:25 DEBUG ----- output ----- 19:53:25 DEBUG ----- output ----- 19:53:25 DEBUG { 19:53:25 DEBUG { 19:53:25 DEBUG "status": "success", 19:53:25 DEBUG "status": "success", 19:53:25 DEBUG "data": { 19:53:25 DEBUG "data": { 19:53:25 DEBUG "resultType": "vector", 19:53:25 DEBUG "resultType": "vector", 19:53:25 DEBUG "result": [ 19:53:25 DEBUG "result": [ 19:53:25 DEBUG { 19:53:25 DEBUG { 19:53:25 DEBUG "metric": { 19:53:25 DEBUG "metric": { 19:53:25 DEBUG "instance": "10.142.0.107:9100" 19:53:25 DEBUG "instance": "10.142.0.107:9100" 19:53:25 DEBUG }, 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG [...] 19:53:25 DEBUG [...] 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG "instance": "10.142.0.43:9100" 19:53:25 DEBUG "instance": "10.142.0.43:9100" 19:53:25 DEBUG }, 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 1713380005, 19:53:25 DEBUG "0.03023333333333298" 19:53:25 DEBUG 1713380005, 19:53:25 DEBUG ] 19:53:25 DEBUG "28616246.566666666" 19:53:25 DEBUG } 19:53:25 DEBUG ] 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG } 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG } 19:53:25 DEBUG } 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380005" 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713380005" 19:53:25 INFO [http_cmd]: http status code OK 19:53:25 DEBUG --- status code --- 19:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:25 DEBUG --- http response --- 19:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:53:25 DEBUG ----- output ----- 19:53:25 DEBUG { 19:53:25 DEBUG "status": "success", 19:53:25 DEBUG "data": { 19:53:25 DEBUG "resultType": "vector", 19:53:25 DEBUG "result": [ 19:53:25 DEBUG { 19:53:25 DEBUG "metric": { 19:53:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG [...] 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 1713380005, 19:53:25 DEBUG "0.0005" 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG } 19:53:25 INFO [http_cmd]: http status code OK 19:53:25 DEBUG --- status code --- 19:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:25 DEBUG --- http response --- 19:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:53:25 DEBUG ----- output ----- 19:53:25 DEBUG { 19:53:25 DEBUG "status": "success", 19:53:25 DEBUG "data": { 19:53:25 DEBUG "resultType": "vector", 19:53:25 DEBUG "result": [ 19:53:25 DEBUG { 19:53:25 DEBUG "metric": { 19:53:25 DEBUG "instance": "10.142.0.107:9100" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG [...] 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG "instance": "10.142.0.43:9100" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 1713380005, 19:53:25 DEBUG "56726156288" 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG } 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380005" 19:53:25 INFO [http_cmd]: http status code OK 19:53:25 DEBUG --- status code --- 19:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:25 DEBUG --- http response --- 19:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:53:25 DEBUG ----- output ----- 19:53:25 DEBUG { 19:53:25 DEBUG "status": "success", 19:53:25 DEBUG "data": { 19:53:25 DEBUG "resultType": "vector", 19:53:25 DEBUG "result": [ 19:53:25 DEBUG { 19:53:25 DEBUG "metric": { 19:53:25 DEBUG "pod": "ds-cts-0" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG [...] 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG "pod": "ds-cts-2" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 1713380005, 19:53:25 DEBUG "0.00014128000000000003" 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG } 19:53:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380005" 19:53:25 INFO [http_cmd]: http status code OK 19:53:25 DEBUG --- status code --- 19:53:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:25 DEBUG --- http response --- 19:53:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:53:25 DEBUG ----- output ----- 19:53:25 DEBUG { 19:53:25 DEBUG "status": "success", 19:53:25 DEBUG "data": { 19:53:25 DEBUG "resultType": "vector", 19:53:25 DEBUG "result": [ 19:53:25 DEBUG { 19:53:25 DEBUG "metric": { 19:53:25 DEBUG "instance": "10.142.0.107:9100" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG [...] 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG 19:53:25 DEBUG "instance": "10.142.0.43:9100" 19:53:25 DEBUG }, 19:53:25 DEBUG "value": [ 19:53:25 DEBUG 1713380005, 19:53:25 DEBUG "28616308.433333334" 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG ] 19:53:25 DEBUG } 19:53:25 DEBUG } 19:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380006" 19:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380006" 19:53:26 INFO [http_cmd]: http status code OK 19:53:26 DEBUG --- status code --- 19:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:26 DEBUG --- http response --- 19:53:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380006" 19:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380006" 19:53:26 INFO [http_cmd]: http status code OK 19:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713380006" 19:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380006" 19:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713380006" 19:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380006" 19:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713380006" 19:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380006" 19:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380006" 19:53:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380006" 19:53:26 DEBUG --- status code --- 19:53:26 INFO [http_cmd]: http status code OK 19:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:26 INFO 19:53:26 DEBUG --- status code --- 19:53:26 INFO [http_cmd]: http status code OK 19:53:26 INFO [http_cmd]: http status code OK 19:53:26 INFO [http_cmd]: http status code OK 19:53:26 DEBUG --- http response --- 19:53:26 INFO [http_cmd]: http status code OK 19:53:26 INFO [loop_until]: kubectl --namespace=xlou top pods 19:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:26 DEBUG --- status code --- 19:53:26 DEBUG --- status code --- 19:53:26 DEBUG --- status code --- 19:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:53:26 INFO [http_cmd]: http status code OK 19:53:26 DEBUG --- status code --- 19:53:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:53:26 DEBUG --- http response --- 19:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:26 DEBUG ----- output ----- 19:53:26 DEBUG --- status code --- 19:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:53:26 INFO [http_cmd]: http status code OK 19:53:26 INFO [http_cmd]: http status code OK 19:53:26 DEBUG --- http response --- 19:53:26 DEBUG --- http response --- 19:53:26 DEBUG --- http response --- 19:53:26 INFO [http_cmd]: http status code OK 19:53:26 DEBUG { 19:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:26 DEBUG --- http response --- 19:53:26 DEBUG ----- output ----- 19:53:26 DEBUG --- status code --- 19:53:26 DEBUG --- status code --- 19:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:53:26 INFO [http_cmd]: http status code OK 19:53:26 DEBUG --- status code --- 19:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:53:26 DEBUG "status": "success", 19:53:26 DEBUG --- http response --- 19:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:53:26 DEBUG { 19:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:26 DEBUG ----- output ----- 19:53:26 DEBUG ----- output ----- 19:53:26 DEBUG --- status code --- 19:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:26 DEBUG ----- output ----- 19:53:26 DEBUG "data": { 19:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:53:26 DEBUG ----- output ----- 19:53:26 DEBUG "status": "success", 19:53:26 DEBUG --- http response --- 19:53:26 DEBUG --- http response --- 19:53:26 DEBUG { 19:53:26 DEBUG { 19:53:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:53:26 DEBUG --- http response --- 19:53:26 DEBUG { 19:53:26 DEBUG "resultType": "vector", 19:53:26 DEBUG ----- output ----- 19:53:26 DEBUG { 19:53:26 DEBUG "data": { 19:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:53:26 DEBUG "status": "success", 19:53:26 DEBUG "status": "success", 19:53:26 DEBUG --- http response --- 19:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:53:26 DEBUG "status": "success", 19:53:26 DEBUG "result": [ 19:53:26 DEBUG { 19:53:26 DEBUG "status": "success", 19:53:26 DEBUG "resultType": "vector", 19:53:26 DEBUG ----- output ----- 19:53:26 DEBUG ----- output ----- 19:53:26 DEBUG "data": { 19:53:26 DEBUG "data": { 19:53:26 DEBUG ----- output ----- 19:53:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:53:26 DEBUG "data": { 19:53:26 DEBUG { 19:53:26 DEBUG "status": "success", 19:53:26 DEBUG "data": { 19:53:26 DEBUG "result": [ 19:53:26 DEBUG { 19:53:26 DEBUG { 19:53:26 DEBUG "resultType": "vector", 19:53:26 DEBUG "resultType": "vector", 19:53:26 DEBUG { 19:53:26 DEBUG ----- output ----- 19:53:26 DEBUG "resultType": "vector", 19:53:26 DEBUG "metric": { 19:53:26 DEBUG "data": { 19:53:26 DEBUG "resultType": "vector", 19:53:26 DEBUG { 19:53:26 DEBUG "status": "success", 19:53:26 DEBUG "status": "success", 19:53:26 DEBUG "result": [ 19:53:26 DEBUG "result": [ 19:53:26 DEBUG "status": "success", 19:53:26 DEBUG { 19:53:26 DEBUG "result": [ 19:53:26 DEBUG "pod": "ds-cts-0" 19:53:26 DEBUG "resultType": "vector", 19:53:26 DEBUG "result": [ 19:53:26 DEBUG "metric": { 19:53:26 DEBUG "data": { 19:53:26 DEBUG "data": { 19:53:26 DEBUG { 19:53:26 DEBUG { 19:53:26 DEBUG "data": { 19:53:26 DEBUG "status": "success", 19:53:26 DEBUG { 19:53:26 DEBUG }, 19:53:26 DEBUG "result": [ 19:53:26 DEBUG { 19:53:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:53:26 DEBUG "resultType": "vector", 19:53:26 DEBUG "resultType": "vector", 19:53:26 DEBUG "metric": { 19:53:26 DEBUG "metric": { 19:53:26 DEBUG "resultType": "vector", 19:53:26 DEBUG "data": { 19:53:26 DEBUG "metric": { 19:53:26 DEBUG "value": [ 19:53:26 DEBUG { 19:53:26 DEBUG "metric": { 19:53:26 DEBUG }, 19:53:26 DEBUG "result": [ 19:53:26 DEBUG "result": [ 19:53:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:53:26 DEBUG "pod": "ds-idrepo-0" 19:53:26 DEBUG "result": [ 19:53:26 DEBUG "resultType": "vector", 19:53:26 DEBUG "pod": "ds-idrepo-0" 19:53:26 DEBUG 19:53:26 DEBUG "metric": { 19:53:26 DEBUG "pod": "ds-idrepo-0" 19:53:26 DEBUG "value": [ 19:53:26 DEBUG { 19:53:26 DEBUG { 19:53:26 DEBUG }, 19:53:26 DEBUG }, 19:53:26 DEBUG { 19:53:26 DEBUG "result": [ 19:53:26 DEBUG }, 19:53:26 DEBUG 19:53:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:53:26 DEBUG }, 19:53:26 DEBUG 19:53:26 DEBUG "metric": { 19:53:26 DEBUG "metric": { 19:53:26 DEBUG "value": [ 19:53:26 DEBUG "value": [ 19:53:26 DEBUG "metric": { 19:53:26 DEBUG { 19:53:26 DEBUG "value": [ 19:53:26 DEBUG 19:53:26 DEBUG }, 19:53:26 DEBUG "value": [ 19:53:26 DEBUG 19:53:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:53:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:53:26 DEBUG "metric": { 19:53:26 DEBUG 19:53:26 DEBUG [...] 19:53:26 DEBUG "value": [ 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG }, 19:53:26 DEBUG }, 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG }, 19:53:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG [...] 19:53:26 DEBUG "value": [ 19:53:26 DEBUG "value": [ 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG "value": [ 19:53:26 DEBUG }, 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG [...] 19:53:26 DEBUG [...] 19:53:26 DEBUG 19:53:26 DEBUG "value": [ 19:53:26 DEBUG [...] 19:53:26 DEBUG 19:53:26 DEBUG "pod": "ds-cts-2" 19:53:26 DEBUG [...] 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG }, 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG [...] 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG [...] 19:53:26 DEBUG "value": [ 19:53:26 DEBUG 19:53:26 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG [...] 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG 1713380006, 19:53:26 DEBUG 19:53:26 DEBUG }, 19:53:26 DEBUG [...] 19:53:26 DEBUG 19:53:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:53:26 DEBUG "pod": "ds-cts-2" 19:53:26 DEBUG 19:53:26 DEBUG [...] 19:53:26 DEBUG "pod": "ds-idrepo-2" 19:53:26 DEBUG 19:53:26 DEBUG "0" 19:53:26 DEBUG "pod": "ds-idrepo-2" 19:53:26 DEBUG "value": [ 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG }, 19:53:26 DEBUG }, 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG }, 19:53:26 DEBUG 19:53:26 DEBUG ] 19:53:26 DEBUG } 19:53:26 DEBUG }, 19:53:26 DEBUG 19:53:26 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:53:26 DEBUG "value": [ 19:53:26 DEBUG "value": [ 19:53:26 DEBUG 19:53:26 DEBUG 19:53:26 DEBUG "value": [ 19:53:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:53:26 DEBUG 1713380006, 19:53:26 DEBUG ] 19:53:26 DEBUG "value": [ 19:53:26 DEBUG 19:53:26 DEBUG }, 19:53:26 DEBUG 1713380006, 19:53:26 DEBUG 1713380006, 19:53:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:53:26 DEBUG 19:53:26 DEBUG 1713380006, 19:53:26 DEBUG }, 19:53:26 DEBUG "8.005915775339767" 19:53:26 DEBUG } 19:53:26 DEBUG 1713380006, 19:53:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:53:26 DEBUG "value": [ 19:53:26 DEBUG "8.027472628823691" 19:53:26 DEBUG "0" 19:53:26 DEBUG }, 19:53:26 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:53:26 DEBUG }, 19:53:26 DEBUG "value": [ 19:53:26 DEBUG ] 19:53:26 DEBUG } 19:53:26 DEBUG "0" 19:53:26 DEBUG }, 19:53:26 DEBUG 1713380006, 19:53:26 DEBUG ] 19:53:26 DEBUG ] 19:53:26 DEBUG "value": [ 19:53:26 DEBUG "0" 19:53:26 DEBUG "value": [ 19:53:26 DEBUG 1713380006, 19:53:26 DEBUG } 19:53:26 DEBUG ] 19:53:26 DEBUG "value": [ 19:53:26 DEBUG "0" 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG 1713380006, 19:53:26 DEBUG ] 19:53:26 DEBUG 1713380006, 19:53:26 DEBUG "123649.5" 19:53:26 DEBUG ] 19:53:26 DEBUG } 19:53:26 DEBUG 1713380006, 19:53:26 DEBUG ] 19:53:26 DEBUG ] 19:53:26 DEBUG ] 19:53:26 DEBUG "1115.5666666666666" 19:53:26 DEBUG } 19:53:26 DEBUG "37675488.10872027" 19:53:26 DEBUG ] 19:53:26 DEBUG } 19:53:26 DEBUG ] 19:53:26 DEBUG "3383599104" 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG ] 19:53:26 INFO 19:53:26 DEBUG ] 19:53:26 DEBUG ] 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG ] 19:53:26 DEBUG ] 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 INFO [loop_until]: kubectl --namespace=xlou top node 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG ] 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG ] 19:53:26 DEBUG } 19:53:26 DEBUG ] 19:53:26 DEBUG } 19:53:26 DEBUG ] 19:53:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 DEBUG } 19:53:26 INFO [loop_until]: OK (rc = 0) 19:53:26 DEBUG --- stdout --- 19:53:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-7996cfcc7c-8h7fs 6477m 5186Mi am-7996cfcc7c-8j9vh 6367m 4989Mi am-7996cfcc7c-989c6 6625m 4844Mi ds-cts-0 3493m 5455Mi ds-cts-1 3429m 5275Mi ds-cts-2 3497m 5555Mi ds-idrepo-0 9572m 2751Mi ds-idrepo-1 322m 1374Mi ds-idrepo-2 294m 1429Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 960Mi idm-b85f9d978-wnhxb 4m 993Mi lodemon-5cd7976d78-6npxn 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7950m 3133Mi 19:53:26 DEBUG --- stderr --- 19:53:26 DEBUG 19:53:26 INFO [loop_until]: OK (rc = 0) 19:53:26 DEBUG --- stdout --- 19:53:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5442Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9174m 30% 5594Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 49m 0% 2556Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 6869m 23% 7804Mi 6% gke-xlou-cdm-default-pool-1b0769c8-w15q 3388m 11% 7600Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5032Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 7330m 24% 7499Mi 6% gke-xlou-cdm-ds-0f601751-cf7b 16638m 55% 3673Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 7342m 24% 6937Mi 6% gke-xlou-cdm-ds-0f601751-j488 371m 1% 3930Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 3372m 11% 8721Mi 7% gke-xlou-cdm-ds-0f601751-nj6q 422m 1% 4388Mi 3% gke-xlou-cdm-ds-0f601751-nmxm 3423m 11% 7050Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8154m 51% 6736Mi 11% 19:53:26 DEBUG --- stderr --- 19:53:26 DEBUG 19:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380065" 19:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380065" 19:54:25 INFO [http_cmd]: http status code OK 19:54:25 DEBUG --- status code --- 19:54:25 INFO [http_cmd]: http status code OK 19:54:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:25 DEBUG --- status code --- 19:54:25 DEBUG --- http response --- 19:54:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:54:25 DEBUG --- http response --- 19:54:25 DEBUG ----- output ----- 19:54:25 DEBUG { 19:54:25 DEBUG "status": "success", 19:54:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:54:25 DEBUG "data": { 19:54:25 DEBUG "resultType": "vector", 19:54:25 DEBUG "result": [ 19:54:25 DEBUG { 19:54:25 DEBUG "metric": { 19:54:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG [...] 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 1713380065, 19:54:25 DEBUG "1135.6666666666667" 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG } 19:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713380065" 19:54:25 ERROR [http_cmd]: ERROR 19:54:25 ERROR --- status code --- 19:54:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:54:25 ERROR --- http response --- 19:54:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:54:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380065" 19:54:25 INFO [http_cmd]: http status code OK 19:54:25 DEBUG --- status code --- 19:54:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:25 DEBUG --- http response --- 19:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:54:25 DEBUG ----- output ----- 19:54:25 DEBUG { 19:54:25 DEBUG "status": "success", 19:54:25 DEBUG "data": { 19:54:25 DEBUG "resultType": "vector", 19:54:25 DEBUG "result": [ 19:54:25 DEBUG { 19:54:25 DEBUG "metric": { 19:54:25 DEBUG "pod": "ds-cts-0" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG [...] 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG "pod": "ds-cts-2" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 1713380065, 19:54:25 DEBUG "0" 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG } 19:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380065" 19:54:25 INFO [http_cmd]: http status code OK 19:54:25 DEBUG --- status code --- 19:54:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:25 DEBUG --- http response --- 19:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:54:25 DEBUG ----- output ----- 19:54:25 DEBUG { 19:54:25 DEBUG "status": "success", 19:54:25 DEBUG "data": { 19:54:25 DEBUG "resultType": "vector", 19:54:25 DEBUG "result": [ 19:54:25 DEBUG { 19:54:25 DEBUG "metric": { 19:54:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG [...] 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 1713380065, 19:54:25 DEBUG "0" 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG } 19:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380065" 19:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380065" 19:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380065" 19:54:25 INFO [http_cmd]: http status code OK 19:54:25 DEBUG --- status code --- 19:54:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:25 DEBUG --- http response --- 19:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:54:25 DEBUG ----- output ----- 19:54:25 DEBUG { 19:54:25 DEBUG "status": "success", 19:54:25 DEBUG "data": { 19:54:25 DEBUG "resultType": "vector", 19:54:25 DEBUG "result": [ 19:54:25 DEBUG { 19:54:25 DEBUG "metric": { 19:54:25 DEBUG "pod": "ds-idrepo-0" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG [...] 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG "pod": "ds-idrepo-2" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 1713380065, 19:54:25 DEBUG "0" 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG } 19:54:25 INFO [http_cmd]: http status code OK 19:54:25 DEBUG --- status code --- 19:54:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:25 DEBUG --- http response --- 19:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:54:25 DEBUG ----- output ----- 19:54:25 DEBUG { 19:54:25 DEBUG "status": "success", 19:54:25 DEBUG "data": { 19:54:25 DEBUG "resultType": "vector", 19:54:25 DEBUG "result": [ 19:54:25 DEBUG { 19:54:25 DEBUG "metric": { 19:54:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG [...] 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 1713380065, 19:54:25 DEBUG "1135.6" 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG } 19:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380065" 19:54:25 INFO [http_cmd]: http status code OK 19:54:25 DEBUG --- status code --- 19:54:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:25 DEBUG --- http response --- 19:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:54:25 DEBUG ----- output ----- 19:54:25 DEBUG { 19:54:25 DEBUG "status": "success", 19:54:25 DEBUG "data": { 19:54:25 DEBUG "resultType": "vector", 19:54:25 DEBUG "result": [ 19:54:25 DEBUG { 19:54:25 DEBUG "metric": { 19:54:25 DEBUG "instance": "10.142.0.107:9100" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG [...] 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG "instance": "10.142.0.43:9100" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 1713380065, 19:54:25 DEBUG "0.023333333333333393" 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG } 19:54:25 INFO [http_cmd]: http status code OK 19:54:25 DEBUG --- status code --- 19:54:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:25 DEBUG --- http response --- 19:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:54:25 DEBUG ----- output ----- 19:54:25 DEBUG { 19:54:25 DEBUG "status": "success", 19:54:25 DEBUG "data": { 19:54:25 DEBUG "resultType": "vector", 19:54:25 DEBUG "result": [ 19:54:25 DEBUG { 19:54:25 DEBUG "metric": { 19:54:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG [...] 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 1713380065, 19:54:25 DEBUG "0" 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG } 19:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380065" 19:54:25 INFO [http_cmd]: http status code OK 19:54:25 DEBUG --- status code --- 19:54:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:25 DEBUG --- http response --- 19:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:54:25 DEBUG ----- output ----- 19:54:25 DEBUG { 19:54:25 DEBUG "status": "success", 19:54:25 DEBUG "data": { 19:54:25 DEBUG "resultType": "vector", 19:54:25 DEBUG "result": [ 19:54:25 DEBUG { 19:54:25 DEBUG "metric": { 19:54:25 DEBUG "instance": "10.142.0.107:9100" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG [...] 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG "instance": "10.142.0.43:9100" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 1713380065, 19:54:25 DEBUG "57976009728" 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG } 19:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380065" 19:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380065" 19:54:25 INFO [http_cmd]: http status code OK 19:54:25 INFO [http_cmd]: http status code OK 19:54:25 DEBUG --- status code --- 19:54:25 DEBUG --- status code --- 19:54:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:25 DEBUG --- http response --- 19:54:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713380065" 19:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:54:25 DEBUG --- http response --- 19:54:25 DEBUG ----- output ----- 19:54:25 DEBUG { 19:54:25 DEBUG "status": "success", 19:54:25 DEBUG "data": { 19:54:25 DEBUG "resultType": "vector", 19:54:25 DEBUG "result": [ 19:54:25 DEBUG { 19:54:25 DEBUG "metric": { 19:54:25 DEBUG "instance": "10.142.0.107:9100" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG [...] 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:54:25 DEBUG 19:54:25 DEBUG ----- output ----- 19:54:25 DEBUG "instance": "10.142.0.43:9100" 19:54:25 DEBUG { 19:54:25 DEBUG }, 19:54:25 DEBUG "status": "success", 19:54:25 DEBUG "value": [ 19:54:25 DEBUG "data": { 19:54:25 DEBUG 1713380065, 19:54:25 DEBUG "resultType": "vector", 19:54:25 DEBUG "0.03090000000000046" 19:54:25 DEBUG "result": [ 19:54:25 DEBUG ] 19:54:25 DEBUG { 19:54:25 DEBUG } 19:54:25 DEBUG "metric": { 19:54:25 DEBUG ] 19:54:25 DEBUG "instance": "10.142.0.107:9100" 19:54:25 DEBUG } 19:54:25 DEBUG }, 19:54:25 DEBUG } 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG [...] 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG "instance": "10.142.0.43:9100" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 1713380065, 19:54:25 DEBUG "28604819.633333333" 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG } 19:54:25 INFO [http_cmd]: http status code OK 19:54:25 DEBUG --- status code --- 19:54:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:25 DEBUG --- http response --- 19:54:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:54:25 DEBUG ----- output ----- 19:54:25 DEBUG { 19:54:25 DEBUG "status": "success", 19:54:25 DEBUG "data": { 19:54:25 DEBUG "resultType": "vector", 19:54:25 DEBUG "result": [ 19:54:25 DEBUG { 19:54:25 DEBUG "metric": { 19:54:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG [...] 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG 19:54:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:54:25 DEBUG }, 19:54:25 DEBUG "value": [ 19:54:25 DEBUG 1713380065, 19:54:25 DEBUG "0.001" 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG ] 19:54:25 DEBUG } 19:54:25 DEBUG } 19:54:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380065" 19:54:26 INFO [http_cmd]: http status code OK 19:54:26 DEBUG --- status code --- 19:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:26 DEBUG --- http response --- 19:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:54:26 DEBUG ----- output ----- 19:54:26 DEBUG { 19:54:26 DEBUG "status": "success", 19:54:26 DEBUG "data": { 19:54:26 DEBUG "resultType": "vector", 19:54:26 DEBUG "result": [ 19:54:26 DEBUG { 19:54:26 DEBUG "metric": { 19:54:26 DEBUG "pod": "ds-cts-0" 19:54:26 DEBUG }, 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG [...] 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG "pod": "ds-cts-2" 19:54:26 DEBUG }, 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 1713380065, 19:54:26 DEBUG "0.000017660000000000004" 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG } 19:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380066" 19:54:26 INFO [http_cmd]: http status code OK 19:54:26 DEBUG --- status code --- 19:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:26 DEBUG --- http response --- 19:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:54:26 DEBUG ----- output ----- 19:54:26 DEBUG { 19:54:26 DEBUG "status": "success", 19:54:26 DEBUG "data": { 19:54:26 DEBUG "resultType": "vector", 19:54:26 DEBUG "result": [ 19:54:26 DEBUG { 19:54:26 DEBUG "metric": { 19:54:26 DEBUG "instance": "10.142.0.107:9100" 19:54:26 DEBUG }, 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG [...] 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG "instance": "10.142.0.43:9100" 19:54:26 DEBUG }, 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 1713380066, 19:54:26 DEBUG "28606782.4" 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG } 19:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380066" 19:54:26 INFO [http_cmd]: http status code OK 19:54:26 DEBUG --- status code --- 19:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:26 DEBUG --- http response --- 19:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (26 lines): 19:54:26 DEBUG ----- output ----- 19:54:26 DEBUG { 19:54:26 DEBUG "status": "success", 19:54:26 DEBUG "data": { 19:54:26 DEBUG "resultType": "vector", 19:54:26 DEBUG "result": [ 19:54:26 DEBUG { 19:54:26 DEBUG "metric": { 19:54:26 DEBUG "pod": "am-7996cfcc7c-8j9vh" 19:54:26 DEBUG }, 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG [...] 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:54:26 DEBUG }, 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 1713380066, 19:54:26 DEBUG "0" 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG } 19:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380066" 19:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380066" 19:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380066" 19:54:26 INFO [http_cmd]: http status code OK 19:54:26 DEBUG --- status code --- 19:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:26 DEBUG --- http response --- 19:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:54:26 DEBUG ----- output ----- 19:54:26 DEBUG { 19:54:26 DEBUG "status": "success", 19:54:26 DEBUG "data": { 19:54:26 DEBUG "resultType": "vector", 19:54:26 DEBUG "result": [ 19:54:26 DEBUG { 19:54:26 DEBUG "metric": { 19:54:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:54:26 DEBUG }, 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG [...] 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:54:26 DEBUG }, 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 1713380066, 19:54:26 DEBUG "0" 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG } 19:54:26 INFO [http_cmd]: http status code OK 19:54:26 DEBUG --- status code --- 19:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:26 DEBUG --- http response --- 19:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:54:26 DEBUG ----- output ----- 19:54:26 DEBUG { 19:54:26 DEBUG "status": "success", 19:54:26 DEBUG "data": { 19:54:26 DEBUG "resultType": "vector", 19:54:26 DEBUG "result": [ 19:54:26 DEBUG { 19:54:26 DEBUG "metric": { 19:54:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:54:26 DEBUG }, 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG [...] 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:54:26 DEBUG }, 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 1713380066, 19:54:26 DEBUG "190648.5" 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG } 19:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380066" 19:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713380066" 19:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380066" 19:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713380066" 19:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713380066" 19:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380066" 19:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380066" 19:54:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380066" 19:54:26 INFO [http_cmd]: http status code OK 19:54:26 DEBUG --- status code --- 19:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:26 DEBUG --- http response --- 19:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:54:26 DEBUG ----- output ----- 19:54:26 DEBUG { 19:54:26 DEBUG "status": "success", 19:54:26 DEBUG "data": { 19:54:26 DEBUG "resultType": "vector", 19:54:26 DEBUG "result": [ 19:54:26 DEBUG { 19:54:26 DEBUG "metric": { 19:54:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:54:26 DEBUG }, 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG [...] 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:54:26 DEBUG }, 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 1713380066, 19:54:26 DEBUG "0" 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG } 19:54:26 INFO [http_cmd]: http status code OK 19:54:26 DEBUG --- status code --- 19:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:26 DEBUG --- http response --- 19:54:26 INFO [http_cmd]: http status code OK 19:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:54:26 INFO [http_cmd]: http status code OK 19:54:26 DEBUG --- status code --- 19:54:26 INFO [http_cmd]: http status code OK 19:54:26 DEBUG ----- output ----- 19:54:26 DEBUG { 19:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:26 DEBUG --- status code --- 19:54:26 DEBUG --- status code --- 19:54:26 DEBUG "status": "success", 19:54:26 DEBUG --- http response --- 19:54:26 INFO [http_cmd]: http status code OK 19:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:26 INFO [http_cmd]: http status code OK 19:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:26 DEBUG "data": { 19:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:54:26 DEBUG --- status code --- 19:54:26 DEBUG --- http response --- 19:54:26 DEBUG --- status code --- 19:54:26 DEBUG --- http response --- 19:54:26 DEBUG "resultType": "vector", 19:54:26 DEBUG ----- output ----- 19:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:26 INFO [http_cmd]: http status code OK 19:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:54:26 DEBUG "result": [ 19:54:26 DEBUG { 19:54:26 DEBUG --- http response --- 19:54:26 DEBUG ----- output ----- 19:54:26 DEBUG --- http response --- 19:54:26 DEBUG --- status code --- 19:54:26 DEBUG ----- output ----- 19:54:26 DEBUG { 19:54:26 DEBUG "status": "success", 19:54:26 DEBUG { 19:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:54:26 DEBUG { 19:54:26 DEBUG "metric": { 19:54:26 DEBUG "data": { 19:54:26 DEBUG "status": "success", 19:54:26 DEBUG ----- output ----- 19:54:26 DEBUG --- http response --- 19:54:26 DEBUG ----- output ----- 19:54:26 DEBUG "status": "success", 19:54:26 DEBUG "pod": "ds-cts-0" 19:54:26 DEBUG "resultType": "vector", 19:54:26 DEBUG "data": { 19:54:26 DEBUG { 19:54:26 DEBUG { 19:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:54:26 DEBUG "data": { 19:54:26 DEBUG }, 19:54:26 DEBUG "result": [ 19:54:26 DEBUG "resultType": "vector", 19:54:26 INFO [http_cmd]: http status code OK 19:54:26 DEBUG "status": "success", 19:54:26 DEBUG "status": "success", 19:54:26 DEBUG ----- output ----- 19:54:26 DEBUG { 19:54:26 DEBUG "value": [ 19:54:26 DEBUG { 19:54:26 DEBUG "result": [ 19:54:26 DEBUG --- status code --- 19:54:26 DEBUG "data": { 19:54:26 DEBUG "data": { 19:54:26 DEBUG "resultType": "vector", 19:54:26 DEBUG "status": "success", 19:54:26 DEBUG 19:54:26 DEBUG "metric": { 19:54:26 DEBUG { 19:54:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:54:26 DEBUG "resultType": "vector", 19:54:26 DEBUG "resultType": "vector", 19:54:26 DEBUG "result": [ 19:54:26 DEBUG "result": [ 19:54:26 DEBUG 19:54:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:54:26 DEBUG "metric": { 19:54:26 DEBUG --- http response --- 19:54:26 DEBUG "result": [ 19:54:26 DEBUG "data": { 19:54:26 DEBUG "resultType": "vector", 19:54:26 DEBUG { 19:54:26 DEBUG 19:54:26 DEBUG }, 19:54:26 DEBUG "pod": "ds-idrepo-0" 19:54:26 DEBUG { 19:54:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:54:26 DEBUG { 19:54:26 DEBUG "result": [ 19:54:26 DEBUG "metric": { 19:54:26 DEBUG [...] 19:54:26 DEBUG "value": [ 19:54:26 DEBUG }, 19:54:26 DEBUG "metric": { 19:54:26 DEBUG ----- output ----- 19:54:26 DEBUG "metric": { 19:54:26 DEBUG { 19:54:26 DEBUG "pod": "ds-idrepo-0" 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG "value": [ 19:54:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:54:26 DEBUG { 19:54:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:54:26 DEBUG "metric": { 19:54:26 DEBUG }, 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG }, 19:54:26 DEBUG "status": "success", 19:54:26 DEBUG }, 19:54:26 DEBUG "pod": "ds-idrepo-0" 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG "value": [ 19:54:26 DEBUG "data": { 19:54:26 DEBUG "value": [ 19:54:26 DEBUG }, 19:54:26 DEBUG 19:54:26 DEBUG "pod": "ds-cts-2" 19:54:26 DEBUG [...] 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG "resultType": "vector", 19:54:26 DEBUG 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 19:54:26 DEBUG }, 19:54:26 DEBUG 19:54:26 DEBUG [...] 19:54:26 DEBUG 19:54:26 DEBUG "result": [ 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG { 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG [...] 19:54:26 DEBUG 1713380066, 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG [...] 19:54:26 DEBUG "metric": { 19:54:26 DEBUG [...] 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG "0" 19:54:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:54:26 DEBUG 19:54:26 DEBUG [...] 19:54:26 DEBUG 19:54:26 DEBUG ] 19:54:26 DEBUG }, 19:54:26 DEBUG "pod": "ds-idrepo-2" 19:54:26 DEBUG 19:54:26 DEBUG }, 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG } 19:54:26 DEBUG "value": [ 19:54:26 DEBUG }, 19:54:26 DEBUG 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 19:54:26 DEBUG 19:54:26 DEBUG "pod": "ds-cts-2" 19:54:26 DEBUG ] 19:54:26 DEBUG 1713380066, 19:54:26 DEBUG "value": [ 19:54:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:54:26 DEBUG 19:54:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:54:26 DEBUG 19:54:26 DEBUG }, 19:54:26 DEBUG } 19:54:26 DEBUG "6265909248" 19:54:26 DEBUG 1713380066, 19:54:26 DEBUG }, 19:54:26 DEBUG 19:54:26 DEBUG }, 19:54:26 DEBUG "pod": "ds-idrepo-2" 19:54:26 DEBUG "value": [ 19:54:26 DEBUG } 19:54:26 DEBUG ] 19:54:26 DEBUG "0" 19:54:26 DEBUG "value": [ 19:54:26 DEBUG 19:54:26 DEBUG "value": [ 19:54:26 DEBUG }, 19:54:26 DEBUG 1713380066, 19:54:26 DEBUG } 19:54:26 DEBUG ] 19:54:26 DEBUG 1713380066, 19:54:26 DEBUG [...] 19:54:26 DEBUG 1713380066, 19:54:26 DEBUG "value": [ 19:54:26 DEBUG "0" 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG "8.034929678875551" 19:54:26 DEBUG 19:54:26 DEBUG "1135.6666666666667" 19:54:26 DEBUG 1713380066, 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG ] 19:54:26 DEBUG ] 19:54:26 DEBUG 19:54:26 DEBUG ] 19:54:26 DEBUG "0" 19:54:26 DEBUG } 19:54:26 DEBUG } 19:54:26 DEBUG } 19:54:26 DEBUG } 19:54:26 DEBUG 19:54:26 DEBUG } 19:54:26 DEBUG ] 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG ] 19:54:26 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG } 19:54:26 DEBUG } 19:54:26 DEBUG }, 19:54:26 DEBUG } 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG } 19:54:26 DEBUG "value": [ 19:54:26 DEBUG } 19:54:26 DEBUG } 19:54:26 DEBUG 1713380066, 19:54:26 DEBUG } 19:54:26 DEBUG "0" 19:54:26 DEBUG ] 19:54:26 DEBUG } 19:54:26 DEBUG ] 19:54:26 INFO 19:54:26 DEBUG } 19:54:26 INFO [loop_until]: kubectl --namespace=xlou top pods 19:54:26 DEBUG } 19:54:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:54:26 INFO 19:54:26 INFO [loop_until]: kubectl --namespace=xlou top node 19:54:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:54:26 INFO [loop_until]: OK (rc = 0) 19:54:26 DEBUG --- stdout --- 19:54:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-7996cfcc7c-8h7fs 6565m 5303Mi am-7996cfcc7c-8j9vh 7251m 7446Mi am-7996cfcc7c-989c6 6145m 4857Mi ds-cts-0 4295m 6282Mi ds-cts-1 3813m 6376Mi ds-cts-2 4273m 6303Mi ds-idrepo-0 8692m 2812Mi ds-idrepo-1 276m 1404Mi ds-idrepo-2 280m 1441Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 6m 960Mi idm-b85f9d978-wnhxb 6m 993Mi lodemon-5cd7976d78-6npxn 10m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7964m 5881Mi 19:54:26 DEBUG --- stderr --- 19:54:26 DEBUG 19:54:26 INFO [loop_until]: OK (rc = 0) 19:54:26 DEBUG --- stdout --- 19:54:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5441Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8918m 29% 5659Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 48m 0% 2545Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 6748m 22% 7812Mi 6% gke-xlou-cdm-default-pool-1b0769c8-w15q 3953m 13% 8438Mi 7% gke-xlou-cdm-default-pool-1b0769c8-wpwl 168m 0% 5027Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 7849m 26% 9964Mi 8% gke-xlou-cdm-ds-0f601751-cf7b 15557m 52% 3748Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 7563m 25% 7046Mi 6% gke-xlou-cdm-ds-0f601751-j488 367m 1% 3947Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 3904m 13% 9818Mi 8% gke-xlou-cdm-ds-0f601751-nj6q 394m 1% 4434Mi 3% gke-xlou-cdm-ds-0f601751-nmxm 4047m 13% 7954Mi 7% gke-xlou-cdm-frontend-a8771548-i82n 8203m 51% 9324Mi 15% 19:54:26 DEBUG --- stderr --- 19:54:26 DEBUG 19:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380125" 19:55:25 INFO [http_cmd]: http status code OK 19:55:25 DEBUG --- status code --- 19:55:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:25 DEBUG --- http response --- 19:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:55:25 DEBUG ----- output ----- 19:55:25 DEBUG { 19:55:25 DEBUG "status": "success", 19:55:25 DEBUG "data": { 19:55:25 DEBUG "resultType": "vector", 19:55:25 DEBUG "result": [ 19:55:25 DEBUG { 19:55:25 DEBUG "metric": { 19:55:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG [...] 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 1713380125, 19:55:25 DEBUG "1151.4" 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG } 19:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380125" 19:55:25 INFO [http_cmd]: http status code OK 19:55:25 DEBUG --- status code --- 19:55:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:25 DEBUG --- http response --- 19:55:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713380125" 19:55:25 ERROR [http_cmd]: ERROR 19:55:25 ERROR --- status code --- 19:55:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:55:25 ERROR --- http response --- 19:55:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:55:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380125" 19:55:25 INFO [http_cmd]: http status code OK 19:55:25 DEBUG --- status code --- 19:55:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:25 DEBUG --- http response --- 19:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:55:25 DEBUG ----- output ----- 19:55:25 DEBUG { 19:55:25 DEBUG "status": "success", 19:55:25 DEBUG "data": { 19:55:25 DEBUG "resultType": "vector", 19:55:25 DEBUG "result": [ 19:55:25 DEBUG { 19:55:25 DEBUG "metric": { 19:55:25 DEBUG "pod": "ds-cts-0" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG [...] 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG "pod": "ds-cts-2" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 1713380125, 19:55:25 DEBUG "0" 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG } 19:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380125" 19:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380125" 19:55:25 INFO [http_cmd]: http status code OK 19:55:25 DEBUG --- status code --- 19:55:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:25 DEBUG --- http response --- 19:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:55:25 DEBUG ----- output ----- 19:55:25 DEBUG { 19:55:25 DEBUG "status": "success", 19:55:25 DEBUG "data": { 19:55:25 DEBUG "resultType": "vector", 19:55:25 DEBUG "result": [ 19:55:25 DEBUG { 19:55:25 DEBUG "metric": { 19:55:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG [...] 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 1713380125, 19:55:25 DEBUG "0" 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG } 19:55:25 INFO [http_cmd]: http status code OK 19:55:25 DEBUG --- status code --- 19:55:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:25 DEBUG --- http response --- 19:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:55:25 DEBUG ----- output ----- 19:55:25 DEBUG { 19:55:25 DEBUG "status": "success", 19:55:25 DEBUG "data": { 19:55:25 DEBUG "resultType": "vector", 19:55:25 DEBUG "result": [ 19:55:25 DEBUG { 19:55:25 DEBUG "metric": { 19:55:25 DEBUG "pod": "ds-idrepo-0" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG [...] 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG "pod": "ds-idrepo-2" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 1713380125, 19:55:25 DEBUG "0" 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG } 19:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380125" 19:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380125" 19:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380125" 19:55:25 INFO [http_cmd]: http status code OK 19:55:25 DEBUG --- status code --- 19:55:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:25 DEBUG --- http response --- 19:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:55:25 DEBUG ----- output ----- 19:55:25 DEBUG { 19:55:25 DEBUG "status": "success", 19:55:25 DEBUG "data": { 19:55:25 DEBUG "resultType": "vector", 19:55:25 DEBUG "result": [ 19:55:25 DEBUG { 19:55:25 DEBUG "metric": { 19:55:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG [...] 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 1713380125, 19:55:25 DEBUG "1151.3" 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG } 19:55:25 INFO [http_cmd]: http status code OK 19:55:25 DEBUG --- status code --- 19:55:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:25 DEBUG --- http response --- 19:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:55:25 DEBUG ----- output ----- 19:55:25 DEBUG { 19:55:25 DEBUG "status": "success", 19:55:25 DEBUG "data": { 19:55:25 DEBUG "resultType": "vector", 19:55:25 DEBUG "result": [ 19:55:25 DEBUG { 19:55:25 DEBUG "metric": { 19:55:25 DEBUG "instance": "10.142.0.107:9100" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG [...] 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG "instance": "10.142.0.43:9100" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 1713380125, 19:55:25 DEBUG "0.02200000000000006" 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG } 19:55:25 INFO [http_cmd]: http status code OK 19:55:25 DEBUG --- status code --- 19:55:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:25 DEBUG --- http response --- 19:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:55:25 DEBUG ----- output ----- 19:55:25 DEBUG { 19:55:25 DEBUG "status": "success", 19:55:25 DEBUG "data": { 19:55:25 DEBUG "resultType": "vector", 19:55:25 DEBUG "result": [ 19:55:25 DEBUG { 19:55:25 DEBUG "metric": { 19:55:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG [...] 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 1713380125, 19:55:25 DEBUG "0" 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG } 19:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380125" 19:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713380125" 19:55:25 INFO [http_cmd]: http status code OK 19:55:25 DEBUG --- status code --- 19:55:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:25 DEBUG --- http response --- 19:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:55:25 DEBUG ----- output ----- 19:55:25 DEBUG { 19:55:25 DEBUG "status": "success", 19:55:25 DEBUG "data": { 19:55:25 DEBUG "resultType": "vector", 19:55:25 DEBUG "result": [ 19:55:25 DEBUG { 19:55:25 DEBUG "metric": { 19:55:25 DEBUG "instance": "10.142.0.107:9100" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG [...] 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG "instance": "10.142.0.43:9100" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 1713380125, 19:55:25 DEBUG "59352534016" 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG } 19:55:25 INFO [http_cmd]: http status code OK 19:55:25 DEBUG --- status code --- 19:55:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:25 DEBUG --- http response --- 19:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:55:25 DEBUG ----- output ----- 19:55:25 DEBUG { 19:55:25 DEBUG "status": "success", 19:55:25 DEBUG "data": { 19:55:25 DEBUG "resultType": "vector", 19:55:25 DEBUG "result": [ 19:55:25 DEBUG { 19:55:25 DEBUG "metric": { 19:55:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG [...] 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG 19:55:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:55:25 DEBUG }, 19:55:25 DEBUG "value": [ 19:55:25 DEBUG 1713380125, 19:55:25 DEBUG "0.0675" 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG ] 19:55:25 DEBUG } 19:55:25 DEBUG } 19:55:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380125" 19:55:25 INFO [http_cmd]: http status code OK 19:55:25 DEBUG --- status code --- 19:55:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:25 DEBUG --- http response --- 19:55:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:55:25 DEBUG ----- output ----- 19:55:25 DEBUG { 19:55:25 DEBUG "status": "success", 19:55:26 DEBUG "data": { 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG "result": [ 19:55:26 DEBUG { 19:55:26 DEBUG "metric": { 19:55:26 DEBUG "instance": "10.142.0.107:9100" 19:55:26 DEBUG }, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG [...] 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG "instance": "10.142.0.43:9100" 19:55:26 DEBUG }, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 1713380125, 19:55:26 DEBUG "39224507.3" 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380126" 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 DEBUG --- status code --- 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG --- http response --- 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:55:26 DEBUG ----- output ----- 19:55:26 DEBUG { 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG "data": { 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG "result": [ 19:55:26 DEBUG { 19:55:26 DEBUG "metric": { 19:55:26 DEBUG "instance": "10.142.0.107:9100" 19:55:26 DEBUG }, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG [...] 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG "instance": "10.142.0.43:9100" 19:55:26 DEBUG }, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG "0.03606666666666716" 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380126" 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 DEBUG --- status code --- 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG --- http response --- 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:55:26 DEBUG ----- output ----- 19:55:26 DEBUG { 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG "data": { 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG "result": [ 19:55:26 DEBUG { 19:55:26 DEBUG "metric": { 19:55:26 DEBUG "pod": "ds-cts-0" 19:55:26 DEBUG }, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG [...] 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG "pod": "ds-cts-2" 19:55:26 DEBUG }, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG "0.00023333333333333333" 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380126" 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 DEBUG --- status code --- 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG --- http response --- 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:55:26 DEBUG ----- output ----- 19:55:26 DEBUG { 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG "data": { 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG "result": [ 19:55:26 DEBUG { 19:55:26 DEBUG "metric": { 19:55:26 DEBUG "instance": "10.142.0.107:9100" 19:55:26 DEBUG }, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG [...] 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG "instance": "10.142.0.43:9100" 19:55:26 DEBUG }, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG "39224871.3" 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380126" 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 DEBUG --- status code --- 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG --- http response --- 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:55:26 DEBUG ----- output ----- 19:55:26 DEBUG { 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG "data": { 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG "result": [ 19:55:26 DEBUG { 19:55:26 DEBUG "metric": { 19:55:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:55:26 DEBUG }, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG [...] 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:55:26 DEBUG }, 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380126" 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380126" 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG "1460.5646916666665" 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380126" 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380126" 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 DEBUG --- status code --- 19:55:26 DEBUG --- status code --- 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG --- http response --- 19:55:26 DEBUG --- http response --- 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:55:26 DEBUG ----- output ----- 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 DEBUG ----- output ----- 19:55:26 DEBUG { 19:55:26 DEBUG --- status code --- 19:55:26 DEBUG { 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG "data": { 19:55:26 DEBUG --- http response --- 19:55:26 DEBUG "data": { 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:55:26 DEBUG "result": [ 19:55:26 DEBUG "result": [ 19:55:26 DEBUG ----- output ----- 19:55:26 DEBUG { 19:55:26 DEBUG { 19:55:26 DEBUG { 19:55:26 DEBUG "metric": { 19:55:26 DEBUG "metric": { 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:55:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:55:26 DEBUG "data": { 19:55:26 DEBUG }, 19:55:26 DEBUG }, 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG "value": [ 19:55:26 DEBUG "value": [ 19:55:26 DEBUG "result": [ 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG { 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG "metric": { 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG "pod": "ds-cts-0" 19:55:26 DEBUG [...] 19:55:26 DEBUG [...] 19:55:26 DEBUG }, 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 19:55:26 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 19:55:26 DEBUG 19:55:26 DEBUG }, 19:55:26 DEBUG }, 19:55:26 DEBUG [...] 19:55:26 DEBUG "value": [ 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG 19:55:26 DEBUG "0.000032574553069378486" 19:55:26 DEBUG "158.8397254430527" 19:55:26 DEBUG 19:55:26 DEBUG ] 19:55:26 DEBUG ] 19:55:26 DEBUG "pod": "ds-cts-2" 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 DEBUG }, 19:55:26 DEBUG ] 19:55:26 DEBUG ] 19:55:26 DEBUG "value": [ 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 DEBUG "0" 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 DEBUG --- status code --- 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG --- http response --- 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:55:26 DEBUG ----- output ----- 19:55:26 DEBUG { 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG "data": { 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG "result": [ 19:55:26 DEBUG { 19:55:26 DEBUG "metric": { 19:55:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:55:26 DEBUG }, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG [...] 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:55:26 DEBUG }, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG "220939" 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713380126" 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380126" 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380126" 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380126" 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713380126" 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713380126" 19:55:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380126" 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 DEBUG --- status code --- 19:55:26 DEBUG --- status code --- 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG --- status code --- 19:55:26 DEBUG --- http response --- 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 DEBUG --- http response --- 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:55:26 DEBUG --- status code --- 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 DEBUG --- http response --- 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:55:26 DEBUG ----- output ----- 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG --- status code --- 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:55:26 DEBUG ----- output ----- 19:55:26 DEBUG { 19:55:26 DEBUG --- status code --- 19:55:26 INFO [http_cmd]: http status code OK 19:55:26 DEBUG --- http response --- 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG ----- output ----- 19:55:26 DEBUG { 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG --- status code --- 19:55:26 DEBUG --- http response --- 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG "data": { 19:55:26 DEBUG --- http response --- 19:55:26 DEBUG { 19:55:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:55:26 DEBUG ----- output ----- 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:55:26 DEBUG "data": { 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:55:26 DEBUG --- http response --- 19:55:26 DEBUG { 19:55:26 DEBUG ----- output ----- 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG "result": [ 19:55:26 DEBUG "data": { 19:55:26 DEBUG ----- output ----- 19:55:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG { 19:55:26 DEBUG "result": [ 19:55:26 DEBUG { 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG { 19:55:26 DEBUG ----- output ----- 19:55:26 DEBUG "data": { 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG "metric": { 19:55:26 DEBUG "result": [ 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG { 19:55:26 DEBUG { 19:55:26 DEBUG "result": [ 19:55:26 DEBUG { 19:55:26 DEBUG "pod": "ds-idrepo-0" 19:55:26 DEBUG { 19:55:26 DEBUG "data": { 19:55:26 DEBUG "status": "success", 19:55:26 DEBUG "data": { 19:55:26 DEBUG "metric": { 19:55:26 DEBUG "metric": { 19:55:26 DEBUG }, 19:55:26 DEBUG "metric": { 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG "data": { 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:55:26 DEBUG }, 19:55:26 DEBUG "pod": "ds-idrepo-0" 19:55:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:55:26 DEBUG "result": [ 19:55:26 DEBUG "resultType": "vector", 19:55:26 DEBUG "result": [ 19:55:26 DEBUG "value": [ 19:55:26 DEBUG "value": [ 19:55:26 DEBUG }, 19:55:26 DEBUG }, 19:55:26 DEBUG { 19:55:26 DEBUG "result": [ 19:55:26 DEBUG { 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG "value": [ 19:55:26 DEBUG "value": [ 19:55:26 DEBUG "metric": { 19:55:26 DEBUG { 19:55:26 DEBUG "metric": { 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:55:26 DEBUG "metric": { 19:55:26 DEBUG "pod": "ds-idrepo-0" 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG }, 19:55:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:55:26 DEBUG }, 19:55:26 DEBUG [...] 19:55:26 DEBUG [...] 19:55:26 DEBUG 19:55:26 DEBUG [...] 19:55:26 DEBUG "value": [ 19:55:26 DEBUG }, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG [...] 19:55:26 DEBUG "pod": "ds-idrepo-2" 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG }, 19:55:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:55:26 DEBUG "pod": "ds-idrepo-2" 19:55:26 DEBUG [...] 19:55:26 DEBUG 19:55:26 DEBUG [...] 19:55:26 DEBUG 19:55:26 DEBUG "value": [ 19:55:26 DEBUG }, 19:55:26 DEBUG }, 19:55:26 DEBUG 19:55:26 DEBUG [...] 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:55:26 DEBUG "0" 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG 19:55:26 DEBUG }, 19:55:26 DEBUG ] 19:55:26 DEBUG "7550369792" 19:55:26 DEBUG "0" 19:55:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:55:26 DEBUG 19:55:26 DEBUG "pod": "ds-cts-2" 19:55:26 DEBUG "value": [ 19:55:26 DEBUG } 19:55:26 DEBUG ] 19:55:26 DEBUG ] 19:55:26 DEBUG }, 19:55:26 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 19:55:26 DEBUG }, 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 DEBUG "value": [ 19:55:26 DEBUG }, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG "1151.4666666666667" 19:55:26 DEBUG } 19:55:26 DEBUG ] 19:55:26 DEBUG ] 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG "value": [ 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 DEBUG "8.026839783186906" 19:55:26 DEBUG 1713380126, 19:55:26 DEBUG "0" 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 DEBUG ] 19:55:26 DEBUG "0" 19:55:26 DEBUG ] 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 DEBUG ] 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 DEBUG } 19:55:26 INFO 19:55:26 INFO [loop_until]: kubectl --namespace=xlou top pods 19:55:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:55:26 INFO 19:55:26 INFO [loop_until]: kubectl --namespace=xlou top node 19:55:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:55:26 INFO [loop_until]: OK (rc = 0) 19:55:26 DEBUG --- stdout --- 19:55:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-7996cfcc7c-8h7fs 6748m 7895Mi am-7996cfcc7c-8j9vh 6663m 7496Mi am-7996cfcc7c-989c6 7077m 7332Mi ds-cts-0 4871m 7126Mi ds-cts-1 3994m 7041Mi ds-cts-2 3908m 6984Mi ds-idrepo-0 8633m 2845Mi ds-idrepo-1 257m 1438Mi ds-idrepo-2 288m 1451Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 5m 960Mi idm-b85f9d978-wnhxb 4m 993Mi lodemon-5cd7976d78-6npxn 10m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 8059m 7106Mi 19:55:26 DEBUG --- stderr --- 19:55:26 DEBUG 19:55:26 INFO [loop_until]: OK (rc = 0) 19:55:26 DEBUG --- stdout --- 19:55:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5441Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8858m 29% 5703Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 49m 0% 2547Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 7683m 25% 10293Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 5888m 19% 9556Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 157m 0% 5047Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 6846m 22% 10015Mi 8% gke-xlou-cdm-ds-0f601751-cf7b 16005m 53% 3720Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 8659m 29% 9659Mi 8% gke-xlou-cdm-ds-0f601751-j488 342m 1% 3959Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 5014m 16% 10813Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 386m 1% 4468Mi 4% gke-xlou-cdm-ds-0f601751-nmxm 5058m 16% 9103Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 8263m 52% 10255Mi 17% 19:55:26 DEBUG --- stderr --- 19:55:26 DEBUG 19:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380185" 19:56:25 INFO [http_cmd]: http status code OK 19:56:25 DEBUG --- status code --- 19:56:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:25 DEBUG --- http response --- 19:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:56:25 DEBUG ----- output ----- 19:56:25 DEBUG { 19:56:25 DEBUG "status": "success", 19:56:25 DEBUG "data": { 19:56:25 DEBUG "resultType": "vector", 19:56:25 DEBUG "result": [ 19:56:25 DEBUG { 19:56:25 DEBUG "metric": { 19:56:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:56:25 DEBUG }, 19:56:25 DEBUG "value": [ 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG [...] 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:56:25 DEBUG }, 19:56:25 DEBUG "value": [ 19:56:25 DEBUG 1713380185, 19:56:25 DEBUG "1020.9" 19:56:25 DEBUG ] 19:56:25 DEBUG } 19:56:25 DEBUG ] 19:56:25 DEBUG } 19:56:25 DEBUG } 19:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380185" 19:56:25 INFO [http_cmd]: http status code OK 19:56:25 DEBUG --- status code --- 19:56:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:25 DEBUG --- http response --- 19:56:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713380185" 19:56:25 ERROR [http_cmd]: ERROR 19:56:25 ERROR --- status code --- 19:56:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:56:25 ERROR --- http response --- 19:56:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:56:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380185" 19:56:25 INFO [http_cmd]: http status code OK 19:56:25 DEBUG --- status code --- 19:56:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:25 DEBUG --- http response --- 19:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:56:25 DEBUG ----- output ----- 19:56:25 DEBUG { 19:56:25 DEBUG "status": "success", 19:56:25 DEBUG "data": { 19:56:25 DEBUG "resultType": "vector", 19:56:25 DEBUG "result": [ 19:56:25 DEBUG { 19:56:25 DEBUG "metric": { 19:56:25 DEBUG "pod": "ds-cts-0" 19:56:25 DEBUG }, 19:56:25 DEBUG "value": [ 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG [...] 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG "pod": "ds-cts-2" 19:56:25 DEBUG }, 19:56:25 DEBUG "value": [ 19:56:25 DEBUG 1713380185, 19:56:25 DEBUG "0" 19:56:25 DEBUG ] 19:56:25 DEBUG } 19:56:25 DEBUG ] 19:56:25 DEBUG } 19:56:25 DEBUG } 19:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380185" 19:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380185" 19:56:25 INFO [http_cmd]: http status code OK 19:56:25 DEBUG --- status code --- 19:56:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:25 DEBUG --- http response --- 19:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:56:25 DEBUG ----- output ----- 19:56:25 DEBUG { 19:56:25 DEBUG "status": "success", 19:56:25 DEBUG "data": { 19:56:25 DEBUG "resultType": "vector", 19:56:25 DEBUG "result": [ 19:56:25 DEBUG { 19:56:25 DEBUG "metric": { 19:56:25 DEBUG "pod": "ds-idrepo-1" 19:56:25 DEBUG }, 19:56:25 DEBUG "value": [ 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG [...] 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:56:25 DEBUG }, 19:56:25 DEBUG "value": [ 19:56:25 DEBUG 1713380185, 19:56:25 DEBUG "0" 19:56:25 DEBUG ] 19:56:25 DEBUG } 19:56:25 DEBUG ] 19:56:25 DEBUG } 19:56:25 DEBUG } 19:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380185" 19:56:25 INFO [http_cmd]: http status code OK 19:56:25 DEBUG --- status code --- 19:56:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:25 DEBUG --- http response --- 19:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:56:25 DEBUG ----- output ----- 19:56:25 DEBUG { 19:56:25 DEBUG "status": "success", 19:56:25 DEBUG "data": { 19:56:25 DEBUG "resultType": "vector", 19:56:25 DEBUG "result": [ 19:56:25 DEBUG { 19:56:25 DEBUG "metric": { 19:56:25 DEBUG "pod": "ds-idrepo-0" 19:56:25 DEBUG }, 19:56:25 DEBUG "value": [ 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG [...] 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG "pod": "ds-idrepo-2" 19:56:25 DEBUG }, 19:56:25 DEBUG "value": [ 19:56:25 DEBUG 1713380185, 19:56:25 DEBUG "0" 19:56:25 DEBUG ] 19:56:25 DEBUG } 19:56:25 DEBUG ] 19:56:25 DEBUG } 19:56:25 DEBUG } 19:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380185" 19:56:25 INFO [http_cmd]: http status code OK 19:56:25 DEBUG --- status code --- 19:56:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:25 DEBUG --- http response --- 19:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:56:25 DEBUG ----- output ----- 19:56:25 DEBUG { 19:56:25 DEBUG "status": "success", 19:56:25 DEBUG "data": { 19:56:25 DEBUG "resultType": "vector", 19:56:25 DEBUG "result": [ 19:56:25 DEBUG { 19:56:25 DEBUG "metric": { 19:56:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:56:25 DEBUG }, 19:56:25 DEBUG "value": [ 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG [...] 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:56:25 DEBUG }, 19:56:25 DEBUG "value": [ 19:56:25 DEBUG 1713380185, 19:56:25 DEBUG "0" 19:56:25 DEBUG ] 19:56:25 DEBUG } 19:56:25 DEBUG ] 19:56:25 DEBUG } 19:56:25 DEBUG } 19:56:25 INFO [http_cmd]: http status code OK 19:56:25 DEBUG --- status code --- 19:56:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:25 DEBUG --- http response --- 19:56:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:56:25 DEBUG ----- output ----- 19:56:25 DEBUG { 19:56:25 DEBUG "status": "success", 19:56:25 DEBUG "data": { 19:56:25 DEBUG "resultType": "vector", 19:56:25 DEBUG "result": [ 19:56:25 DEBUG { 19:56:25 DEBUG "metric": { 19:56:25 DEBUG "instance": "10.142.0.107:9100" 19:56:25 DEBUG }, 19:56:25 DEBUG "value": [ 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG [...] 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG 19:56:25 DEBUG "instance": "10.142.0.43:9100" 19:56:25 DEBUG }, 19:56:25 DEBUG "value": [ 19:56:25 DEBUG 1713380185, 19:56:25 DEBUG "0.030333333333333354" 19:56:25 DEBUG ] 19:56:25 DEBUG } 19:56:25 DEBUG ] 19:56:25 DEBUG } 19:56:25 DEBUG } 19:56:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380185" 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380185, 19:56:26 DEBUG "1020.9" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713380186" 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380186" 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "1.722" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380186" 19:56:26 DEBUG ] 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG } 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG } 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "instance": "10.142.0.107:9100" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "instance": "10.142.0.43:9100" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "0.0438333333333342" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "instance": "10.142.0.107:9100" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "instance": "10.142.0.43:9100" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "61154315264" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380186" 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "instance": "10.142.0.107:9100" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "instance": "10.142.0.43:9100" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "39090215.16666667" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380186" 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "instance": "10.142.0.107:9100" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "instance": "10.142.0.43:9100" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "39091587.53333333" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380186" 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "pod": "ds-cts-0" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "pod": "ds-cts-2" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "0.00007286222222222221" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380186" 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "1716.2419916666668" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380186" 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380186" 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380186" 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380186" 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "pod": "ds-idrepo-1" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "0" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "pod": "ds-cts-0" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "pod": "ds-cts-2" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "0" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "pod": "ds-idrepo-1" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "0" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "203442" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380186" 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "pod": "ds-idrepo-0" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "pod": "ds-idrepo-2" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "0" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380186" 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713380186" 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713380186" 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "pod": "ds-idrepo-0" 19:56:26 DEBUG "pod": "ds-idrepo-0" 19:56:26 DEBUG }, 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "pod": "ds-cts-2" 19:56:26 DEBUG "pod": "ds-idrepo-2" 19:56:26 DEBUG }, 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "0" 19:56:26 DEBUG "0" 19:56:26 DEBUG ] 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "8292016128" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380186" 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "1020.9" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713380186" 19:56:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380186" 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG --- http response --- 19:56:26 INFO [http_cmd]: http status code OK 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:56:26 DEBUG --- status code --- 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:56:26 DEBUG { 19:56:26 DEBUG --- http response --- 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "8.031071934446942" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:56:26 DEBUG ----- output ----- 19:56:26 DEBUG { 19:56:26 DEBUG "status": "success", 19:56:26 DEBUG "data": { 19:56:26 DEBUG "resultType": "vector", 19:56:26 DEBUG "result": [ 19:56:26 DEBUG { 19:56:26 DEBUG "metric": { 19:56:26 DEBUG "pod": "ds-idrepo-1" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG [...] 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG 19:56:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:56:26 DEBUG }, 19:56:26 DEBUG "value": [ 19:56:26 DEBUG 1713380186, 19:56:26 DEBUG "0" 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG ] 19:56:26 DEBUG } 19:56:26 DEBUG } 19:56:26 INFO 19:56:26 INFO [loop_until]: kubectl --namespace=xlou top pods 19:56:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:56:26 INFO [loop_until]: OK (rc = 0) 19:56:26 DEBUG --- stdout --- 19:56:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-7996cfcc7c-8h7fs 8398m 8134Mi am-7996cfcc7c-8j9vh 7655m 7551Mi am-7996cfcc7c-989c6 8186m 7523Mi ds-cts-0 8016m 7596Mi ds-cts-1 7625m 7484Mi ds-cts-2 7839m 7750Mi ds-idrepo-0 9383m 2939Mi ds-idrepo-1 273m 1445Mi ds-idrepo-2 268m 1460Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 6m 961Mi idm-b85f9d978-wnhxb 4m 993Mi lodemon-5cd7976d78-6npxn 2m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 7819m 7641Mi 19:56:26 DEBUG --- stderr --- 19:56:26 DEBUG 19:56:26 INFO 19:56:26 INFO [loop_until]: kubectl --namespace=xlou top node 19:56:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:56:26 INFO [loop_until]: OK (rc = 0) 19:56:26 DEBUG --- stdout --- 19:56:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5442Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9612m 32% 5801Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 49m 0% 2544Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 8796m 29% 10482Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 8062m 27% 9721Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 5034Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 8412m 28% 10047Mi 8% gke-xlou-cdm-ds-0f601751-cf7b 14892m 49% 3686Mi 3% gke-xlou-cdm-ds-0f601751-hmm7 9034m 30% 9894Mi 8% gke-xlou-cdm-ds-0f601751-j488 321m 1% 3959Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 7867m 26% 10901Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 380m 1% 4477Mi 4% gke-xlou-cdm-ds-0f601751-nmxm 7957m 26% 9321Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 8191m 51% 10970Mi 18% 19:56:26 DEBUG --- stderr --- 19:56:26 DEBUG 19:56:36 WARNING No values available for selected timeframe. 19:56:36 WARNING No values available for selected timeframe. 19:56:36 WARNING No values available for selected timeframe. 19:56:36 WARNING File /tmp/lodemon_data-20240417_194420/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [17/Apr/2024 19:56:36] "GET /monitoring/average?start_time=20240417_185306&stop_time=20240417_185435 HTTP/1.1" 200 - 19:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380245" 19:57:25 INFO [http_cmd]: http status code OK 19:57:25 DEBUG --- status code --- 19:57:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:25 DEBUG --- http response --- 19:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:57:25 DEBUG ----- output ----- 19:57:25 DEBUG { 19:57:25 DEBUG "status": "success", 19:57:25 DEBUG "data": { 19:57:25 DEBUG "resultType": "vector", 19:57:25 DEBUG "result": [ 19:57:25 DEBUG { 19:57:25 DEBUG "metric": { 19:57:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:57:25 DEBUG }, 19:57:25 DEBUG "value": [ 19:57:25 DEBUG 19:57:25 DEBUG 19:57:25 DEBUG 19:57:25 DEBUG [...] 19:57:25 DEBUG 19:57:25 DEBUG 19:57:25 DEBUG 19:57:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:57:25 DEBUG }, 19:57:25 DEBUG "value": [ 19:57:25 DEBUG 1713380245, 19:57:25 DEBUG "0" 19:57:25 DEBUG ] 19:57:25 DEBUG } 19:57:25 DEBUG ] 19:57:25 DEBUG } 19:57:25 DEBUG } 19:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380245" 19:57:25 INFO [http_cmd]: http status code OK 19:57:25 DEBUG --- status code --- 19:57:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:25 DEBUG --- http response --- 19:57:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713380245" 19:57:25 ERROR [http_cmd]: ERROR 19:57:25 ERROR --- status code --- 19:57:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:57:25 ERROR --- http response --- 19:57:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:57:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:57:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380245" 19:57:25 INFO [http_cmd]: http status code OK 19:57:25 DEBUG --- status code --- 19:57:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:25 DEBUG --- http response --- 19:57:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:57:25 DEBUG ----- output ----- 19:57:25 DEBUG { 19:57:25 DEBUG "status": "success", 19:57:25 DEBUG "data": { 19:57:25 DEBUG "resultType": "vector", 19:57:25 DEBUG "result": [ 19:57:25 DEBUG { 19:57:25 DEBUG "metric": { 19:57:25 DEBUG "pod": "ds-cts-0" 19:57:25 DEBUG }, 19:57:25 DEBUG "value": [ 19:57:25 DEBUG 19:57:25 DEBUG 19:57:25 DEBUG 19:57:25 DEBUG [...] 19:57:25 DEBUG 19:57:25 DEBUG 19:57:25 DEBUG 19:57:25 DEBUG "pod": "ds-cts-2" 19:57:25 DEBUG }, 19:57:25 DEBUG "value": [ 19:57:25 DEBUG 1713380245, 19:57:25 DEBUG "0" 19:57:25 DEBUG ] 19:57:25 DEBUG } 19:57:25 DEBUG ] 19:57:25 DEBUG } 19:57:25 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "am-7996cfcc7c-8j9vh" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "626.3236362246263" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "ds-idrepo-0" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "ds-idrepo-2" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "0" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "0" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "instance": "10.142.0.107:9100" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "instance": "10.142.0.43:9100" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "0.10066666666666672" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380246" 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "instance": "10.142.0.107:9100" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "instance": "10.142.0.43:9100" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "0.15846666666666684" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "0" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "5.0915" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "instance": "10.142.0.107:9100" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "instance": "10.142.0.43:9100" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "14507690.233333332" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "instance": "10.142.0.107:9100" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "instance": "10.142.0.43:9100" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "62837953536" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "instance": "10.142.0.107:9100" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "instance": "10.142.0.43:9100" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "14512781.7" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "ds-cts-0" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "ds-cts-2" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "0.00013952888888888888" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "2135.3" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "150350.5" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "ds-cts-0" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "ds-cts-2" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "0" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "0" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "overseer-0-5c67bf94db-77mh9" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "0" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "ds-idrepo-0" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "ds-idrepo-2" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "0" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "ds-idrepo-0" 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "pod": "ds-idrepo-0" 19:57:26 DEBUG }, 19:57:26 DEBUG "data": { 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG "result": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG { 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "metric": { 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG }, 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "ds-idrepo-2" 19:57:26 DEBUG 19:57:26 DEBUG "pod": "ds-cts-2" 19:57:26 DEBUG }, 19:57:26 DEBUG 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG [...] 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "0" 19:57:26 DEBUG 19:57:26 DEBUG "0" 19:57:26 DEBUG ] 19:57:26 DEBUG 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG }, 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG "value": [ 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG } 19:57:26 DEBUG "117362688" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "0" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713380246" 19:57:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380246" 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "pod": "am-7996cfcc7c-8j9vh" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "0" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO [http_cmd]: http status code OK 19:57:26 DEBUG --- status code --- 19:57:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:57:26 DEBUG --- http response --- 19:57:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:57:26 DEBUG ----- output ----- 19:57:26 DEBUG { 19:57:26 DEBUG "status": "success", 19:57:26 DEBUG "data": { 19:57:26 DEBUG "resultType": "vector", 19:57:26 DEBUG "result": [ 19:57:26 DEBUG { 19:57:26 DEBUG "metric": { 19:57:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG [...] 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG 19:57:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:57:26 DEBUG }, 19:57:26 DEBUG "value": [ 19:57:26 DEBUG 1713380246, 19:57:26 DEBUG "0.03161930875886161" 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG ] 19:57:26 DEBUG } 19:57:26 DEBUG } 19:57:26 INFO 19:57:26 INFO [loop_until]: kubectl --namespace=xlou top pods 19:57:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:57:26 INFO [loop_until]: OK (rc = 0) 19:57:26 DEBUG --- stdout --- 19:57:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-7996cfcc7c-8h7fs 2570m 8125Mi am-7996cfcc7c-8j9vh 2551m 7689Mi am-7996cfcc7c-989c6 2605m 7556Mi ds-cts-0 5955m 7573Mi ds-cts-1 5623m 7442Mi ds-cts-2 6006m 7736Mi ds-idrepo-0 1714m 2939Mi ds-idrepo-1 17m 1451Mi ds-idrepo-2 56m 1462Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 4m 960Mi idm-b85f9d978-wnhxb 4m 992Mi lodemon-5cd7976d78-6npxn 9m 63Mi login-ui-5fdf49fc66-dlhjd 1m 5Mi overseer-0-5c67bf94db-77mh9 8m 16Mi 19:57:26 DEBUG --- stderr --- 19:57:26 DEBUG 19:57:26 INFO 19:57:26 INFO [loop_until]: kubectl --namespace=xlou top node 19:57:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:57:26 INFO [loop_until]: OK (rc = 0) 19:57:26 DEBUG --- stdout --- 19:57:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5433Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1872m 6% 5804Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 46m 0% 2547Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 2673m 8% 10561Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 5831m 19% 9721Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 169m 0% 5039Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2690m 9% 10262Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 67m 0% 2937Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2874m 9% 9963Mi 8% gke-xlou-cdm-ds-0f601751-j488 65m 0% 3953Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 5755m 19% 10900Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 91m 0% 4461Mi 3% gke-xlou-cdm-ds-0f601751-nmxm 6101m 20% 9323Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 99m 0% 3838Mi 6% 19:57:26 DEBUG --- stderr --- 19:57:26 DEBUG 19:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380305" 19:58:25 INFO [http_cmd]: http status code OK 19:58:25 DEBUG --- status code --- 19:58:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:25 DEBUG --- http response --- 19:58:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:58:25 DEBUG ----- output ----- 19:58:25 DEBUG { 19:58:25 DEBUG "status": "success", 19:58:25 DEBUG "data": { 19:58:25 DEBUG "resultType": "vector", 19:58:25 DEBUG "result": [ 19:58:25 DEBUG { 19:58:25 DEBUG "metric": { 19:58:25 DEBUG "pod": "am-7996cfcc7c-989c6" 19:58:25 DEBUG }, 19:58:25 DEBUG "value": [ 19:58:25 DEBUG 19:58:25 DEBUG 19:58:25 DEBUG 19:58:25 DEBUG [...] 19:58:25 DEBUG 19:58:25 DEBUG 19:58:25 DEBUG 19:58:25 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:58:25 DEBUG }, 19:58:25 DEBUG "value": [ 19:58:25 DEBUG 1713380305, 19:58:25 DEBUG "0" 19:58:25 DEBUG ] 19:58:25 DEBUG } 19:58:25 DEBUG ] 19:58:25 DEBUG } 19:58:25 DEBUG } 19:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713380305" 19:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380305" 19:58:25 INFO [http_cmd]: http status code OK 19:58:25 DEBUG --- status code --- 19:58:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:25 DEBUG --- http response --- 19:58:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:58:25 ERROR [http_cmd]: ERROR 19:58:25 ERROR --- status code --- 19:58:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:58:25 ERROR --- http response --- 19:58:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:58:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:58:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380305" 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "ds-cts-0" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "ds-cts-2" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380305, 19:58:26 DEBUG "0" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380306" 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380306" 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "ds-idrepo-0" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "ds-idrepo-2" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "59" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "0" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "instance": "10.142.0.107:9100" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "instance": "10.142.0.43:9100" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "0.0325333333333333" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG } 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG } 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "0" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "instance": "10.142.0.107:9100" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "instance": "10.142.0.43:9100" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "0.0246666666666668" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "ds-idrepo-1" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "8967.399856326754" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380306" 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "8.546" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "instance": "10.142.0.107:9100" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "instance": "10.142.0.43:9100" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "14369892.566666666" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380306" 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "instance": "10.142.0.107:9100" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "instance": "10.142.0.43:9100" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "63951633408" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713380306" 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "ds-cts-0" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "ds-cts-2" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "0.00011976444444444445" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "instance": "10.142.0.107:9100" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "instance": "10.142.0.43:9100" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "14371159.666666668" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "2150.9666666666667" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "ds-cts-0" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "ds-cts-2" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "0" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "81784.5" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "ds-idrepo-0" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "ds-idrepo-2" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "8.133333333333333" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "0" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "ds-idrepo-0" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "ds-idrepo-2" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "6.266666666666667" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "login-ui-5fdf49fc66-dlhjd" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "0" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "ds-idrepo-0" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "ds-cts-2" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "0" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713380306" 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "445812736" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "am-7996cfcc7c-989c6" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "am-7996cfcc7c-8h7fs" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "0" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713380306" 19:58:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713380306" 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "pod": "admin-ui-c4c7bf584-tlgkc" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "pod": "ds-idrepo-1" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "0" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO [http_cmd]: http status code OK 19:58:26 DEBUG --- status code --- 19:58:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:58:26 DEBUG --- http response --- 19:58:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:58:26 DEBUG ----- output ----- 19:58:26 DEBUG { 19:58:26 DEBUG "status": "success", 19:58:26 DEBUG "data": { 19:58:26 DEBUG "resultType": "vector", 19:58:26 DEBUG "result": [ 19:58:26 DEBUG { 19:58:26 DEBUG "metric": { 19:58:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-cf7b" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG [...] 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG 19:58:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:58:26 DEBUG }, 19:58:26 DEBUG "value": [ 19:58:26 DEBUG 1713380306, 19:58:26 DEBUG "0.34062808272350925" 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG ] 19:58:26 DEBUG } 19:58:26 DEBUG } 19:58:26 INFO 19:58:26 INFO [loop_until]: kubectl --namespace=xlou top pods 19:58:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:58:26 INFO [loop_until]: OK (rc = 0) 19:58:26 DEBUG --- stdout --- 19:58:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-tlgkc 1m 9Mi am-7996cfcc7c-8h7fs 2585m 8066Mi am-7996cfcc7c-8j9vh 2493m 7569Mi am-7996cfcc7c-989c6 2658m 7549Mi ds-cts-0 6032m 7580Mi ds-cts-1 5582m 7409Mi ds-cts-2 5913m 7709Mi ds-idrepo-0 2547m 2984Mi ds-idrepo-1 401m 1487Mi ds-idrepo-2 334m 1480Mi end-user-ui-58878d7668-z8s6j 1m 5Mi idm-b85f9d978-txpwv 20m 964Mi idm-b85f9d978-wnhxb 5m 1002Mi lodemon-5cd7976d78-6npxn 8m 63Mi login-ui-5fdf49fc66-dlhjd 1m 4Mi overseer-0-5c67bf94db-77mh9 857m 331Mi 19:58:26 DEBUG --- stderr --- 19:58:26 DEBUG 19:58:26 INFO 19:58:26 INFO [loop_until]: kubectl --namespace=xlou top node 19:58:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:58:26 INFO [loop_until]: OK (rc = 0) 19:58:26 DEBUG --- stdout --- 19:58:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5435Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2586m 8% 5910Mi 5% gke-xlou-cdm-default-pool-1b0769c8-nhlf 47m 0% 2545Mi 2% gke-xlou-cdm-default-pool-1b0769c8-qkns 2671m 8% 10539Mi 9% gke-xlou-cdm-default-pool-1b0769c8-w15q 5872m 19% 9720Mi 8% gke-xlou-cdm-default-pool-1b0769c8-wpwl 167m 0% 5032Mi 4% gke-xlou-cdm-default-pool-1b0769c8-xnmp 2642m 8% 10104Mi 9% gke-xlou-cdm-ds-0f601751-cf7b 62m 0% 2897Mi 2% gke-xlou-cdm-ds-0f601751-hmm7 2721m 9% 9856Mi 8% gke-xlou-cdm-ds-0f601751-j488 454m 1% 3939Mi 3% gke-xlou-cdm-ds-0f601751-mc5d 5615m 18% 10879Mi 9% gke-xlou-cdm-ds-0f601751-nj6q 569m 1% 4438Mi 3% gke-xlou-cdm-ds-0f601751-nmxm 6102m 20% 9315Mi 8% gke-xlou-cdm-frontend-a8771548-i82n 925m 5% 3973Mi 6% 19:58:26 DEBUG --- stderr --- 19:58:26 DEBUG 19:58:27 INFO Finished: True 19:58:27 INFO Waiting for threads to register finish flag 19:59:26 INFO Done. Have a nice day! :) 127.0.0.1 - - [17/Apr/2024 19:59:26] "GET /monitoring/stop HTTP/1.1" 200 - 19:59:29 WARNING File /tmp/lodemon_data-20240417_194420/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6821', ...) 127.0.0.1 - - [17/Apr/2024 19:59:34] "GET /monitoring/process HTTP/1.1" 200 -