==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-84dfb5c75c-qnqlj Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Sun, 14 Apr 2024 02:57:23 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=84dfb5c75c skaffold.dev/run-id=7de071a0-3407-4b94-8a9f-6024c484fe40 Annotations: Status: Running IP: 10.106.46.137 IPs: IP: 10.106.46.137 Controlled By: ReplicaSet/lodemon-84dfb5c75c Containers: lodemon: Container ID: containerd://7f09ad02c128ce2958ce1a16488f9e8c4fc8c1a97a15a0de4e051c0c5196631c Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:06c53901427b11990b989f0a16f422c2c89c055d3393605a0d2d532652d23afc Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Sun, 14 Apr 2024 02:57:24 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6w9gz (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-6w9gz: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 21m default-scheduler Successfully assigned xlou/lodemon-84dfb5c75c-qnqlj to gke-xlou-cdm-ds-0f601751-mc5d Normal Pulling 21m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 21m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 221.245011ms (221.251787ms including waiting) Normal Created 21m kubelet Created container lodemon Normal Started 21m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.46.137:8080 Press CTRL+C to quit 03:58:21 INFO 03:58:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:21 INFO [loop_until]: OK (rc = 0) 03:58:21 DEBUG --- stdout --- 03:58:21 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:21 DEBUG --- stderr --- 03:58:21 DEBUG 03:58:21 INFO 03:58:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:21 INFO [loop_until]: OK (rc = 0) 03:58:21 DEBUG --- stdout --- 03:58:21 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:21 DEBUG --- stderr --- 03:58:21 DEBUG 03:58:21 INFO 03:58:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:21 INFO [loop_until]: OK (rc = 0) 03:58:21 DEBUG --- stdout --- 03:58:21 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:21 DEBUG --- stderr --- 03:58:21 DEBUG 03:58:21 INFO 03:58:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:21 INFO [loop_until]: OK (rc = 0) 03:58:21 DEBUG --- stdout --- 03:58:21 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:21 DEBUG --- stderr --- 03:58:21 DEBUG 03:58:21 INFO 03:58:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:21 INFO [loop_until]: OK (rc = 0) 03:58:21 DEBUG --- stdout --- 03:58:21 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:21 DEBUG --- stderr --- 03:58:21 DEBUG 03:58:21 INFO 03:58:21 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:21 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:21 INFO [loop_until]: OK (rc = 0) 03:58:21 DEBUG --- stdout --- 03:58:21 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:21 DEBUG --- stderr --- 03:58:21 DEBUG 03:58:22 INFO 03:58:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:22 INFO [loop_until]: OK (rc = 0) 03:58:22 DEBUG --- stdout --- 03:58:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:22 DEBUG --- stderr --- 03:58:22 DEBUG 03:58:22 INFO 03:58:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:22 INFO [loop_until]: OK (rc = 0) 03:58:22 DEBUG --- stdout --- 03:58:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:22 DEBUG --- stderr --- 03:58:22 DEBUG 03:58:22 INFO 03:58:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:22 INFO [loop_until]: OK (rc = 0) 03:58:22 DEBUG --- stdout --- 03:58:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:22 DEBUG --- stderr --- 03:58:22 DEBUG 03:58:22 INFO 03:58:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:22 INFO [loop_until]: OK (rc = 0) 03:58:22 DEBUG --- stdout --- 03:58:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:22 DEBUG --- stderr --- 03:58:22 DEBUG 03:58:22 INFO 03:58:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:22 INFO [loop_until]: OK (rc = 0) 03:58:22 DEBUG --- stdout --- 03:58:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:22 DEBUG --- stderr --- 03:58:22 DEBUG 03:58:22 INFO 03:58:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:22 INFO [loop_until]: OK (rc = 0) 03:58:22 DEBUG --- stdout --- 03:58:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:22 DEBUG --- stderr --- 03:58:22 DEBUG 03:58:22 INFO 03:58:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:22 INFO [loop_until]: OK (rc = 0) 03:58:22 DEBUG --- stdout --- 03:58:22 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:22 DEBUG --- stderr --- 03:58:22 DEBUG 03:58:22 INFO 03:58:22 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:22 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:23 INFO [loop_until]: OK (rc = 0) 03:58:23 DEBUG --- stdout --- 03:58:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:23 DEBUG --- stderr --- 03:58:23 DEBUG 03:58:23 INFO 03:58:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:23 INFO [loop_until]: OK (rc = 0) 03:58:23 DEBUG --- stdout --- 03:58:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:23 DEBUG --- stderr --- 03:58:23 DEBUG 03:58:23 INFO 03:58:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:23 INFO [loop_until]: OK (rc = 0) 03:58:23 DEBUG --- stdout --- 03:58:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:23 DEBUG --- stderr --- 03:58:23 DEBUG 03:58:23 INFO 03:58:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:23 INFO [loop_until]: OK (rc = 0) 03:58:23 DEBUG --- stdout --- 03:58:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:23 DEBUG --- stderr --- 03:58:23 DEBUG 03:58:23 INFO 03:58:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:23 INFO [loop_until]: OK (rc = 0) 03:58:23 DEBUG --- stdout --- 03:58:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:23 DEBUG --- stderr --- 03:58:23 DEBUG 03:58:23 INFO 03:58:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:23 INFO [loop_until]: OK (rc = 0) 03:58:23 DEBUG --- stdout --- 03:58:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:23 DEBUG --- stderr --- 03:58:23 DEBUG 03:58:23 INFO 03:58:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:23 INFO [loop_until]: OK (rc = 0) 03:58:23 DEBUG --- stdout --- 03:58:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:23 DEBUG --- stderr --- 03:58:23 DEBUG 03:58:23 INFO 03:58:23 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:23 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:23 INFO [loop_until]: OK (rc = 0) 03:58:23 DEBUG --- stdout --- 03:58:23 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:23 DEBUG --- stderr --- 03:58:23 DEBUG 03:58:24 INFO 03:58:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:24 INFO [loop_until]: OK (rc = 0) 03:58:24 DEBUG --- stdout --- 03:58:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:24 DEBUG --- stderr --- 03:58:24 DEBUG 03:58:24 INFO 03:58:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:24 INFO [loop_until]: OK (rc = 0) 03:58:24 DEBUG --- stdout --- 03:58:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:24 DEBUG --- stderr --- 03:58:24 DEBUG 03:58:24 INFO 03:58:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:24 INFO [loop_until]: OK (rc = 0) 03:58:24 DEBUG --- stdout --- 03:58:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:24 DEBUG --- stderr --- 03:58:24 DEBUG 03:58:24 INFO 03:58:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:24 INFO [loop_until]: OK (rc = 0) 03:58:24 DEBUG --- stdout --- 03:58:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:24 DEBUG --- stderr --- 03:58:24 DEBUG 03:58:24 INFO 03:58:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:24 INFO [loop_until]: OK (rc = 0) 03:58:24 DEBUG --- stdout --- 03:58:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:24 DEBUG --- stderr --- 03:58:24 DEBUG 03:58:24 INFO 03:58:24 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:58:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:24 INFO [loop_until]: OK (rc = 0) 03:58:24 DEBUG --- stdout --- 03:58:24 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:58:24 DEBUG --- stderr --- 03:58:24 DEBUG 03:58:24 INFO Initializing monitoring instance threads 03:58:24 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 03:58:24 INFO Starting instance threads 03:58:24 INFO 03:58:24 INFO Thread started 03:58:24 INFO [loop_until]: kubectl --namespace=xlou top node 03:58:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:24 INFO 03:58:24 INFO Thread started 03:58:24 INFO [loop_until]: kubectl --namespace=xlou top pods 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO Thread started 03:58:24 DEBUG --- status code --- 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG --- status code --- 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO Thread started 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG --- status code --- 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 DEBUG --- status code --- 03:58:24 INFO Thread started 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 INFO Thread started 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG --- http response --- 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 DEBUG ----- output ----- 03:58:24 INFO Thread started 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:58:24 DEBUG { 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 DEBUG ----- output ----- 03:58:24 INFO Thread started 03:58:24 DEBUG { 03:58:24 INFO [loop_until]: OK (rc = 0) 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG { 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG ----- output ----- 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG { 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG --- stdout --- 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG { 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG { 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG "data": { 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "data": { 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 DEBUG "data": { 03:58:24 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8907Mi am-7d597cf7d6-8vr8f 7m 8541Mi am-7d597cf7d6-9wz9h 4m 8450Mi ds-cts-0 14m 7161Mi ds-cts-1 14m 7899Mi ds-cts-2 13m 8111Mi ds-idrepo-0 11m 4581Mi ds-idrepo-1 9m 4295Mi ds-idrepo-2 11m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 991Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 1m 54Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 223Mi 03:58:24 DEBUG "data": { 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "data": { 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG "data": { 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG --- stderr --- 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG "data": { 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG "data": { 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 DEBUG "data": { 03:58:24 DEBUG "data": { 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:58:24 INFO [loop_until]: OK (rc = 0) 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG "data": { 03:58:24 DEBUG "data": { 03:58:24 DEBUG "data": { 03:58:24 DEBUG "data": { 03:58:24 DEBUG "result": [ 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:58:24 DEBUG "result": [ 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 DEBUG "result": [ 03:58:24 DEBUG 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713063504" 03:58:24 DEBUG "result": [ 03:58:24 INFO Thread started 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG "result": [ 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG "result": [ 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG --- stdout --- 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG { 03:58:24 DEBUG "result": [ 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG { 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG "result": [ 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 DEBUG { 03:58:24 DEBUG "result": [ 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4821Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 69m 0% 10275Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 49m 0% 10986Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 84m 0% 3057Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 67m 0% 6330Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 156m 0% 10662Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2914Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 60m 0% 9977Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 5495Mi 4% gke-xlou-cdm-ds-0f601751-j488 65m 0% 9382Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 73m 0% 2331Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 85m 0% 9994Mi 8% gke-xlou-cdm-ds-0f601751-nmxm 59m 0% 6039Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 68m 0% 4029Mi 6% 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG ----- output ----- 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 DEBUG "result": [ 03:58:24 DEBUG "result": [ 03:58:24 DEBUG "result": [ 03:58:24 DEBUG "result": [ 03:58:24 DEBUG "metric": { 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG "metric": { 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG "metric": { 03:58:24 DEBUG "metric": { 03:58:24 DEBUG { 03:58:24 DEBUG "metric": { 03:58:24 DEBUG { 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063504" 03:58:24 DEBUG --- http response --- 03:58:24 INFO Thread started 03:58:24 DEBUG "result": [ 03:58:24 DEBUG "metric": { 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG --- stderr --- 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:58:24 DEBUG "metric": { 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:58:24 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:58:24 DEBUG "metric": { 03:58:24 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:58:24 DEBUG "metric": { 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:58:24 DEBUG ----- output ----- 03:58:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063504" 03:58:24 INFO Thread started 03:58:24 DEBUG "pod": "ds-idrepo-1" 03:58:24 DEBUG "data": { 03:58:24 DEBUG 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG "metric": { 03:58:24 DEBUG "metric": { 03:58:24 DEBUG "metric": { 03:58:24 ERROR [http_cmd]: ERROR 03:58:24 DEBUG "metric": { 03:58:24 DEBUG }, 03:58:24 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:58:24 DEBUG "data": { 03:58:24 DEBUG }, 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:58:24 DEBUG }, 03:58:24 DEBUG }, 03:58:24 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:58:24 DEBUG }, 03:58:24 DEBUG "pod": "ds-idrepo-1" 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 INFO All threads has been started 127.0.0.1 - - [14/Apr/2024 03:58:24] "GET /monitoring/start HTTP/1.1" 200 - 03:58:24 DEBUG }, 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:58:24 DEBUG "data": { 03:58:24 DEBUG "data": { 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG "pod": "ds-idrepo-1" 03:58:24 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:58:24 DEBUG "instance": "10.142.0.107:9100" 03:58:24 ERROR --- status code --- 03:58:24 DEBUG "instance": "10.142.0.107:9100" 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 DEBUG "value": [ 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG }, 03:58:24 DEBUG "value": [ 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "value": [ 03:58:24 DEBUG }, 03:58:24 DEBUG "value": [ 03:58:24 DEBUG }, 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG "metric": { 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "result": [ 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:58:24 DEBUG }, 03:58:24 DEBUG }, 03:58:24 DEBUG "value": [ 03:58:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:58:24 DEBUG }, 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 03:58:24 DEBUG { 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "instance": "10.142.0.107:9100" 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:58:24 DEBUG "data": { 03:58:24 DEBUG 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "result": [ 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG { 03:58:24 DEBUG }, 03:58:24 DEBUG 03:58:24 ERROR --- http response --- 03:58:24 DEBUG "value": [ 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG 03:58:24 DEBUG "result": [ 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG }, 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG { 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG "result": [ 03:58:24 DEBUG "result": [ 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:58:24 DEBUG 03:58:24 DEBUG { 03:58:24 DEBUG "metric": { 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:58:24 DEBUG 03:58:24 DEBUG "data": { 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "metric": { 03:58:24 DEBUG "pod": "ds-idrepo-1" 03:58:24 DEBUG { 03:58:24 DEBUG "metric": { 03:58:24 DEBUG "metric": { 03:58:24 DEBUG "data": { 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:58:24 INFO [http_cmd]: http status code OK 03:58:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG [...] 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG [...] 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "data": { 03:58:24 DEBUG "data": { 03:58:24 DEBUG }, 03:58:24 DEBUG "pod": "ds-idrepo-1" 03:58:24 DEBUG "metric": { 03:58:24 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:58:24 DEBUG 03:58:24 DEBUG "result": [ 03:58:24 DEBUG 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG --- status code --- 03:58:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:58:24 DEBUG 03:58:24 DEBUG }, 03:58:24 DEBUG { 03:58:24 DEBUG 03:58:24 DEBUG "result": [ 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG [...] 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG "value": [ 03:58:24 DEBUG }, 03:58:24 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:58:24 DEBUG }, 03:58:24 DEBUG 03:58:24 DEBUG { 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG [...] 03:58:24 DEBUG --- http response --- 03:58:24 DEBUG 03:58:24 DEBUG [...] 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG 03:58:24 DEBUG { 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "result": [ 03:58:24 DEBUG "result": [ 03:58:24 DEBUG 03:58:24 DEBUG "value": [ 03:58:24 DEBUG }, 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 03:58:24 DEBUG "metric": { 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "data": { 03:58:24 DEBUG 03:58:24 DEBUG "metric": { 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG { 03:58:24 DEBUG { 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG ----- output ----- 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 03:58:24 DEBUG "pod": "ds-idrepo-1" 03:58:24 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 03:58:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:58:24 DEBUG 03:58:24 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:58:24 DEBUG }, 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "metric": { 03:58:24 DEBUG "metric": { 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG [...] 03:58:24 DEBUG }, 03:58:24 DEBUG 03:58:24 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:58:24 DEBUG 03:58:24 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 03:58:24 DEBUG { 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "result": [ 03:58:24 DEBUG }, 03:58:24 DEBUG }, 03:58:24 DEBUG }, 03:58:24 DEBUG }, 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 03:58:24 DEBUG "pod": "ds-idrepo-0" 03:58:24 DEBUG "pod": "ds-idrepo-1" 03:58:24 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG }, 03:58:24 DEBUG "instance": "10.142.0.43:9100" 03:58:24 DEBUG }, 03:58:24 DEBUG "status": "success", 03:58:24 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:58:24 DEBUG [...] 03:58:24 DEBUG { 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "value": [ 03:58:24 DEBUG }, 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG 03:58:24 DEBUG }, 03:58:24 DEBUG }, 03:58:24 DEBUG }, 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "instance": "10.142.0.43:9100" 03:58:24 DEBUG "value": [ 03:58:24 DEBUG }, 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "data": { 03:58:24 DEBUG }, 03:58:24 DEBUG 03:58:24 DEBUG "metric": { 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG "0" 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "pod": "ds-idrepo-0" 03:58:24 DEBUG "0" 03:58:24 DEBUG "instance": "10.142.0.43:9100" 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:58:24 DEBUG 03:58:24 DEBUG }, 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG "resultType": "vector", 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 03:58:24 DEBUG "instance": "10.142.0.107:9100" 03:58:24 DEBUG "0.00021621205131223012" 03:58:24 DEBUG 03:58:24 DEBUG "0.029924979261601033" 03:58:24 DEBUG ] 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG }, 03:58:24 DEBUG ] 03:58:24 DEBUG }, 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG "pod": "ds-idrepo-0" 03:58:24 DEBUG }, 03:58:24 DEBUG 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "0" 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG "0" 03:58:24 DEBUG "result": [ 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG 03:58:24 DEBUG }, 03:58:24 DEBUG ] 03:58:24 DEBUG 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG "0" 03:58:24 DEBUG "value": [ 03:58:24 DEBUG } 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "0" 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "pod": "ds-idrepo-0" 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG }, 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG ] 03:58:24 DEBUG "0" 03:58:24 DEBUG ] 03:58:24 DEBUG { 03:58:24 DEBUG "332296192" 03:58:24 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:58:24 DEBUG "value": [ 03:58:24 DEBUG } 03:58:24 DEBUG [...] 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG ] 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG ] 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG ] 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG }, 03:58:24 DEBUG 03:58:24 DEBUG "pod": "ds-idrepo-0" 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG [...] 03:58:24 DEBUG "4245" 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG "metric": { 03:58:24 DEBUG ] 03:58:24 DEBUG }, 03:58:24 DEBUG 03:58:24 DEBUG ] 03:58:24 DEBUG 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG "0" 03:58:24 DEBUG } 03:58:24 DEBUG "5889.1" 03:58:24 DEBUG } 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 03:58:24 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG "0" 03:58:24 DEBUG 03:58:24 DEBUG ] 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG "instance": "10.142.0.107:9100" 03:58:24 DEBUG } 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG ] 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG }, 03:58:24 DEBUG }, 03:58:24 DEBUG "0" 03:58:24 DEBUG ] 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG }, 03:58:24 DEBUG ] 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG "0" 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "value": [ 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:58:24 DEBUG }, 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG "value": [ 03:58:24 DEBUG } 03:58:24 DEBUG "0" 03:58:24 DEBUG [...] 03:58:24 DEBUG 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG 03:58:24 DEBUG "pod": "ds-cts-1" 03:58:24 DEBUG ] 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG "value": [ 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG 03:58:24 DEBUG 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG 03:58:24 DEBUG }, 03:58:24 DEBUG } 03:58:24 DEBUG "0" 03:58:24 DEBUG ] 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG 03:58:24 DEBUG } 03:58:24 DEBUG "pod": "ds-cts-1" 03:58:24 DEBUG 03:58:24 DEBUG } 03:58:24 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:58:24 DEBUG "value": [ 03:58:24 DEBUG ] 03:58:24 DEBUG "0" 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG "0.015" 03:58:24 DEBUG 03:58:24 DEBUG ] 03:58:24 DEBUG }, 03:58:24 DEBUG "instance": "10.142.0.43:9100" 03:58:24 DEBUG } 03:58:24 DEBUG }, 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG [...] 03:58:24 DEBUG } 03:58:24 DEBUG "value": [ 03:58:24 DEBUG }, 03:58:24 DEBUG "value": [ 03:58:24 DEBUG "0" 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG 03:58:24 DEBUG } 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG "value": [ 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG 03:58:24 DEBUG "0" 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG "1" 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG 03:58:24 DEBUG ] 03:58:24 DEBUG "12679900160" 03:58:24 DEBUG ] 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG "instance": "10.142.0.43:9100" 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG }, 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG "value": [ 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG 1713063504, 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG } 03:58:24 DEBUG "0.000333333333333267" 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG ] 03:58:24 DEBUG } 03:58:24 DEBUG } 03:59:24 INFO 03:59:24 INFO [loop_until]: kubectl --namespace=xlou top node 03:59:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 INFO 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 INFO [loop_until]: kubectl --namespace=xlou top pods 03:59:24 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 ERROR [http_cmd]: ERROR 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 DEBUG --- status code --- 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 ERROR --- status code --- 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 DEBUG --- status code --- 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:24 DEBUG --- status code --- 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 DEBUG --- status code --- 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 DEBUG --- status code --- 03:59:24 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG --- http response --- 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:24 ERROR --- http response --- 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:24 DEBUG --- http response --- 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:24 DEBUG --- http response --- 03:59:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:24 DEBUG --- http response --- 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:24 DEBUG --- http response --- 03:59:24 DEBUG --- http response --- 03:59:24 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:59:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 03:59:24 DEBUG --- http response --- 03:59:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:59:24 DEBUG ----- output ----- 03:59:24 DEBUG --- http response --- 03:59:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:59:24 DEBUG --- http response --- 03:59:24 DEBUG --- http response --- 03:59:24 DEBUG --- http response --- 03:59:24 DEBUG --- http response --- 03:59:24 DEBUG --- http response --- 03:59:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:59:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:59:24 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:59:24 DEBUG ----- output ----- 03:59:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:59:24 DEBUG ----- output ----- 03:59:24 DEBUG { 03:59:24 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:59:24 DEBUG ----- output ----- 03:59:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:59:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:59:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:59:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:59:24 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:59:24 DEBUG ----- output ----- 03:59:24 DEBUG ----- output ----- 03:59:24 DEBUG { 03:59:24 DEBUG ----- output ----- 03:59:24 DEBUG { 03:59:24 DEBUG "status": "success", 03:59:24 DEBUG { 03:59:24 DEBUG ----- output ----- 03:59:24 DEBUG ----- output ----- 03:59:24 DEBUG ----- output ----- 03:59:24 DEBUG ----- output ----- 03:59:24 DEBUG ----- output ----- 03:59:24 DEBUG { 03:59:24 DEBUG { 03:59:24 DEBUG "status": "success", 03:59:24 DEBUG { 03:59:24 DEBUG "status": "success", 03:59:24 DEBUG "data": { 03:59:24 DEBUG "status": "success", 03:59:24 DEBUG { 03:59:24 DEBUG { 03:59:24 DEBUG { 03:59:24 DEBUG { 03:59:24 DEBUG { 03:59:24 DEBUG "status": "success", 03:59:24 DEBUG "status": "success", 03:59:24 DEBUG "data": { 03:59:24 DEBUG "status": "success", 03:59:24 DEBUG "data": { 03:59:24 DEBUG "resultType": "vector", 03:59:24 DEBUG "data": { 03:59:24 DEBUG "status": "success", 03:59:24 DEBUG "status": "success", 03:59:24 DEBUG "status": "success", 03:59:24 DEBUG "status": "success", 03:59:24 DEBUG "status": "success", 03:59:24 DEBUG "data": { 03:59:24 DEBUG "data": { 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063564" 03:59:24 DEBUG "resultType": "vector", 03:59:24 DEBUG "data": { 03:59:24 DEBUG "resultType": "vector", 03:59:24 DEBUG "result": [ 03:59:24 DEBUG "resultType": "vector", 03:59:24 DEBUG "data": { 03:59:24 DEBUG "data": { 03:59:24 DEBUG "data": { 03:59:24 DEBUG "data": { 03:59:24 DEBUG "data": { 03:59:24 DEBUG "resultType": "vector", 03:59:24 DEBUG "resultType": "vector", 03:59:24 DEBUG "result": [ 03:59:24 DEBUG "resultType": "vector", 03:59:24 DEBUG "result": [ 03:59:24 DEBUG { 03:59:24 DEBUG "result": [ 03:59:24 DEBUG "resultType": "vector", 03:59:24 DEBUG "resultType": "vector", 03:59:24 DEBUG "resultType": "vector", 03:59:24 DEBUG "resultType": "vector", 03:59:24 DEBUG "resultType": "vector", 03:59:24 DEBUG "result": [ 03:59:24 DEBUG "result": [ 03:59:24 DEBUG { 03:59:24 DEBUG "result": [ 03:59:24 DEBUG { 03:59:24 DEBUG "metric": { 03:59:24 DEBUG { 03:59:24 DEBUG "result": [ 03:59:24 DEBUG "result": [ 03:59:24 DEBUG "result": [ 03:59:24 DEBUG "result": [ 03:59:24 DEBUG "result": [ 03:59:24 DEBUG { 03:59:24 DEBUG { 03:59:24 DEBUG "metric": { 03:59:24 DEBUG { 03:59:24 DEBUG "metric": { 03:59:24 DEBUG "pod": "ds-idrepo-1" 03:59:24 DEBUG }, 03:59:24 DEBUG { 03:59:24 DEBUG { 03:59:24 DEBUG { 03:59:24 DEBUG { 03:59:24 DEBUG { 03:59:24 DEBUG "metric": { 03:59:24 DEBUG "metric": { 03:59:24 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:59:24 DEBUG "metric": { 03:59:24 DEBUG "instance": "10.142.0.107:9100" 03:59:24 DEBUG "metric": { 03:59:24 DEBUG "value": [ 03:59:24 DEBUG "metric": { 03:59:24 DEBUG "metric": { 03:59:24 DEBUG "metric": { 03:59:24 DEBUG "metric": { 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713063564" 03:59:24 DEBUG "metric": { 03:59:24 DEBUG "pod": "ds-idrepo-1" 03:59:24 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:59:24 DEBUG }, 03:59:24 DEBUG "instance": "10.142.0.107:9100" 03:59:24 DEBUG }, 03:59:24 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:59:24 DEBUG 03:59:24 DEBUG "pod": "ds-idrepo-1" 03:59:24 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:59:24 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:59:24 DEBUG "pod": "ds-idrepo-1" 03:59:24 DEBUG "instance": "10.142.0.107:9100" 03:59:24 DEBUG }, 03:59:24 DEBUG }, 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 DEBUG "value": [ 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 DEBUG }, 03:59:24 DEBUG "value": [ 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 DEBUG }, 03:59:24 DEBUG 03:59:24 DEBUG }, 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063564" 03:59:24 DEBUG }, 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 DEBUG }, 03:59:24 DEBUG }, 03:59:24 DEBUG }, 03:59:24 DEBUG "value": [ 03:59:24 DEBUG 03:59:24 DEBUG 03:59:24 DEBUG "value": [ 03:59:24 DEBUG 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 DEBUG "value": [ 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 DEBUG 03:59:24 DEBUG "value": [ 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713063564" 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 DEBUG "value": [ 03:59:24 INFO [loop_until]: OK (rc = 0) 03:59:24 DEBUG "value": [ 03:59:24 DEBUG "value": [ 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 DEBUG "value": [ 03:59:24 DEBUG "value": [ 03:59:24 DEBUG 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 DEBUG 03:59:24 DEBUG 03:59:24 DEBUG 03:59:24 DEBUG 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 INFO [http_cmd]: http status code OK 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG [...] 03:59:24 DEBUG 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG 03:59:24 DEBUG --- stdout --- 03:59:24 DEBUG 03:59:24 DEBUG 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG 03:59:24 DEBUG 03:59:24 DEBUG 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG 03:59:24 DEBUG 03:59:24 DEBUG 03:59:24 DEBUG 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG --- status code --- 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:24 DEBUG 03:59:25 DEBUG 03:59:24 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:25 DEBUG 03:59:25 INFO [http_cmd]: http status code OK 03:59:25 DEBUG 03:59:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4819Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 193m 0% 10297Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 56m 0% 10974Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3058Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 288m 0% 6377Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 350m 1% 10603Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 75m 0% 2914Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 61m 0% 9986Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 240m 0% 5449Mi 4% gke-xlou-cdm-ds-0f601751-j488 246m 0% 9348Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2203Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 158m 0% 9997Mi 8% gke-xlou-cdm-ds-0f601751-nmxm 563m 1% 6117Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 285m 1% 4033Mi 6% 03:59:25 DEBUG 03:59:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG [...] 03:59:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:25 INFO [loop_until]: OK (rc = 0) 03:59:25 DEBUG [...] 03:59:25 DEBUG 03:59:25 DEBUG [...] 03:59:25 DEBUG 03:59:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:25 DEBUG --- http response --- 03:59:24 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG --- http response --- 03:59:25 DEBUG 03:59:25 DEBUG --- status code --- 03:59:25 DEBUG 03:59:25 DEBUG --- stderr --- 03:59:25 DEBUG 03:59:25 INFO [http_cmd]: http status code OK 03:59:25 DEBUG --- http response --- 03:59:25 DEBUG --- http response --- 03:59:25 DEBUG --- http response --- 03:59:25 DEBUG --- http response --- 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG --- http response --- 03:59:25 DEBUG --- stdout --- 03:59:25 DEBUG 03:59:25 DEBUG [...] 03:59:25 DEBUG 03:59:25 DEBUG [...] 03:59:25 DEBUG --- http response --- 03:59:25 DEBUG --- http response --- 03:59:25 DEBUG 03:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:59:25 DEBUG "pod": "ds-idrepo-0" 03:59:25 INFO [http_cmd]: http status code OK 03:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:59:25 DEBUG [...] 03:59:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:25 DEBUG [...] 03:59:25 DEBUG 03:59:25 DEBUG [...] 03:59:25 DEBUG --- status code --- 03:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:59:25 DEBUG [...] 03:59:25 INFO [http_cmd]: http status code OK 03:59:25 DEBUG [...] 03:59:25 DEBUG 03:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:59:25 DEBUG 03:59:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 7m 8907Mi am-7d597cf7d6-8vr8f 11m 8546Mi am-7d597cf7d6-9wz9h 70m 8469Mi ds-cts-0 174m 7162Mi ds-cts-1 213m 7900Mi ds-cts-2 169m 8112Mi ds-idrepo-0 389m 4584Mi ds-idrepo-1 224m 4302Mi ds-idrepo-2 157m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 17m 997Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 133m 61Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 195m 223Mi 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:59:25 DEBUG [...] 03:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:59:25 DEBUG ----- output ----- 03:59:25 DEBUG }, 03:59:25 DEBUG --- status code --- 03:59:25 DEBUG ----- output ----- 03:59:25 DEBUG 03:59:25 DEBUG --- http response --- 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:25 DEBUG ----- output ----- 03:59:25 DEBUG ----- output ----- 03:59:25 DEBUG ----- output ----- 03:59:25 DEBUG ----- output ----- 03:59:25 DEBUG 03:59:25 DEBUG --- status code --- 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG ----- output ----- 03:59:25 DEBUG 03:59:25 DEBUG --- stderr --- 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG ----- output ----- 03:59:25 DEBUG 03:59:25 DEBUG ----- output ----- 03:59:25 DEBUG { 03:59:25 DEBUG "value": [ 03:59:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:25 DEBUG { 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 03:59:25 DEBUG --- http response --- 03:59:25 DEBUG { 03:59:25 DEBUG { 03:59:25 DEBUG { 03:59:25 DEBUG { 03:59:25 DEBUG 03:59:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:59:25 DEBUG 03:59:25 DEBUG "pod": "ds-idrepo-0" 03:59:25 DEBUG { 03:59:25 DEBUG "pod": "am-7d597cf7d6-8vr8f" 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG "instance": "10.142.0.43:9100" 03:59:25 DEBUG 03:59:25 DEBUG { 03:59:25 DEBUG 03:59:25 DEBUG { 03:59:25 DEBUG "status": "success", 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG --- http response --- 03:59:25 DEBUG "status": "success", 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG ----- output ----- 03:59:25 DEBUG "status": "success", 03:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:59:25 DEBUG "status": "success", 03:59:25 DEBUG "status": "success", 03:59:25 DEBUG "status": "success", 03:59:25 DEBUG 03:59:25 DEBUG --- http response --- 03:59:25 DEBUG 03:59:25 DEBUG }, 03:59:25 DEBUG "status": "success", 03:59:25 DEBUG }, 03:59:25 DEBUG "instance": "10.142.0.43:9100" 03:59:25 DEBUG }, 03:59:25 DEBUG "pod": "am-7d597cf7d6-8vr8f" 03:59:25 DEBUG "status": "success", 03:59:25 DEBUG 03:59:25 DEBUG "status": "success", 03:59:25 DEBUG "data": { 03:59:25 DEBUG "0" 03:59:25 DEBUG "data": { 03:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:59:25 DEBUG "pod": "am-7d597cf7d6-8vr8f" 03:59:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:59:25 DEBUG "pod": "ds-idrepo-0" 03:59:25 DEBUG { 03:59:25 DEBUG "data": { 03:59:25 DEBUG ----- output ----- 03:59:25 DEBUG "data": { 03:59:25 DEBUG "data": { 03:59:25 DEBUG "data": { 03:59:25 DEBUG "instance": "10.142.0.43:9100" 03:59:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:59:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "data": { 03:59:25 DEBUG "value": [ 03:59:25 DEBUG }, 03:59:25 DEBUG "value": [ 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG }, 03:59:25 DEBUG "pod": "ds-cts-1" 03:59:25 DEBUG "data": { 03:59:25 DEBUG "resultType": "vector", 03:59:25 DEBUG ] 03:59:25 DEBUG "resultType": "vector", 03:59:25 DEBUG ----- output ----- 03:59:25 DEBUG }, 03:59:25 DEBUG }, 03:59:25 DEBUG }, 03:59:25 DEBUG "status": "success", 03:59:25 DEBUG "resultType": "vector", 03:59:25 DEBUG { 03:59:25 DEBUG "resultType": "vector", 03:59:25 DEBUG "resultType": "vector", 03:59:25 DEBUG "resultType": "vector", 03:59:25 DEBUG }, 03:59:25 DEBUG }, 03:59:25 DEBUG ----- output ----- 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG "resultType": "vector", 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "data": { 03:59:25 DEBUG "0.0009333333333332897" 03:59:25 DEBUG "value": [ 03:59:25 DEBUG }, 03:59:25 DEBUG "resultType": "vector", 03:59:25 DEBUG "result": [ 03:59:25 DEBUG } 03:59:25 DEBUG "result": [ 03:59:25 DEBUG { 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "data": { 03:59:25 DEBUG "result": [ 03:59:25 DEBUG "status": "success", 03:59:25 DEBUG "result": [ 03:59:25 DEBUG "result": [ 03:59:25 DEBUG "result": [ 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "value": [ 03:59:25 DEBUG { 03:59:25 DEBUG "0" 03:59:25 DEBUG "result": [ 03:59:25 DEBUG "0" 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG "resultType": "vector", 03:59:25 DEBUG ] 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "result": [ 03:59:25 DEBUG { 03:59:25 DEBUG ] 03:59:25 DEBUG { 03:59:25 DEBUG "status": "success", 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG "resultType": "vector", 03:59:25 DEBUG { 03:59:25 DEBUG "data": { 03:59:25 DEBUG { 03:59:25 DEBUG { 03:59:25 DEBUG { 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG "status": "success", 03:59:25 DEBUG ] 03:59:25 DEBUG { 03:59:25 DEBUG ] 03:59:25 DEBUG "2850.5333333333333" 03:59:25 DEBUG "result": [ 03:59:25 DEBUG } 03:59:25 DEBUG "25262.798596491226" 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG { 03:59:25 DEBUG "metric": { 03:59:25 DEBUG } 03:59:25 DEBUG "metric": { 03:59:25 DEBUG "data": { 03:59:25 DEBUG "0.004746713505130863" 03:59:25 DEBUG "333889536" 03:59:25 DEBUG "0" 03:59:25 DEBUG "result": [ 03:59:25 DEBUG "metric": { 03:59:25 DEBUG "resultType": "vector", 03:59:25 DEBUG "metric": { 03:59:25 DEBUG "metric": { 03:59:25 DEBUG "metric": { 03:59:25 DEBUG "5171.266666666667" 03:59:25 DEBUG "0.015" 03:59:25 DEBUG "data": { 03:59:25 DEBUG } 03:59:25 DEBUG "metric": { 03:59:25 DEBUG } 03:59:25 DEBUG ] 03:59:25 DEBUG { 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG "0" 03:59:25 DEBUG "metric": { 03:59:25 DEBUG "pod": "ds-idrepo-1" 03:59:25 DEBUG } 03:59:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:59:25 DEBUG "resultType": "vector", 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG { 03:59:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:59:25 DEBUG "result": [ 03:59:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:59:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:59:25 DEBUG "pod": "ds-idrepo-1" 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG "resultType": "vector", 03:59:25 DEBUG ] 03:59:25 DEBUG "instance": "10.142.0.107:9100" 03:59:25 DEBUG ] 03:59:25 DEBUG } 03:59:25 DEBUG "metric": { 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG ] 03:59:25 DEBUG "instance": "10.142.0.107:9100" 03:59:25 DEBUG }, 03:59:25 DEBUG }, 03:59:25 DEBUG "result": [ 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG "metric": { 03:59:25 DEBUG }, 03:59:25 DEBUG { 03:59:25 DEBUG }, 03:59:25 DEBUG }, 03:59:25 DEBUG }, 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG "result": [ 03:59:25 DEBUG } 03:59:25 DEBUG }, 03:59:25 DEBUG } 03:59:25 DEBUG ] 03:59:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:59:25 DEBUG } 03:59:25 DEBUG ] 03:59:25 DEBUG } 03:59:25 DEBUG }, 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "value": [ 03:59:25 DEBUG { 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "metric": { 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "value": [ 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG { 03:59:25 DEBUG } 03:59:25 DEBUG "value": [ 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG }, 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG "value": [ 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG "metric": { 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG }, 03:59:25 DEBUG 03:59:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG "metric": { 03:59:25 DEBUG 03:59:25 DEBUG } 03:59:25 DEBUG "value": [ 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG "value": [ 03:59:25 DEBUG 03:59:25 DEBUG }, 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG "pod": "ds-idrepo-1" 03:59:25 DEBUG 03:59:25 DEBUG ] 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG [...] 03:59:25 DEBUG }, 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG "value": [ 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG [...] 03:59:25 DEBUG }, 03:59:25 DEBUG 03:59:25 DEBUG } 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG "value": [ 03:59:25 DEBUG 03:59:25 DEBUG [...] 03:59:25 DEBUG 03:59:25 DEBUG [...] 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG "value": [ 03:59:25 DEBUG [...] 03:59:25 DEBUG } 03:59:25 DEBUG [...] 03:59:25 DEBUG [...] 03:59:25 DEBUG [...] 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG [...] 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG [...] 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG "pod": "ds-idrepo-0" 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG [...] 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG }, 03:59:25 DEBUG [...] 03:59:25 DEBUG 03:59:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:59:25 DEBUG 03:59:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:59:25 DEBUG 03:59:25 DEBUG }, 03:59:25 DEBUG [...] 03:59:25 DEBUG "instance": "10.142.0.43:9100" 03:59:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:59:25 DEBUG "instance": "10.142.0.43:9100" 03:59:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:59:25 DEBUG "value": [ 03:59:25 DEBUG 03:59:25 DEBUG 03:59:25 DEBUG }, 03:59:25 DEBUG 03:59:25 DEBUG }, 03:59:25 DEBUG "pod": "ds-idrepo-0" 03:59:25 DEBUG "value": [ 03:59:25 DEBUG 03:59:25 DEBUG }, 03:59:25 DEBUG }, 03:59:25 DEBUG }, 03:59:25 DEBUG }, 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG 03:59:25 DEBUG "pod": "am-7d597cf7d6-8vr8f" 03:59:25 DEBUG "value": [ 03:59:25 DEBUG 03:59:25 DEBUG "value": [ 03:59:25 DEBUG }, 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "0" 03:59:25 DEBUG 03:59:25 DEBUG }, 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "0" 03:59:25 DEBUG 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG ] 03:59:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "0" 03:59:25 DEBUG }, 03:59:25 DEBUG "0" 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG ] 03:59:25 DEBUG "pod": "ds-cts-1" 03:59:25 DEBUG "0.00033333333333333365" 03:59:25 DEBUG "0" 03:59:25 DEBUG "12683041792" 03:59:25 DEBUG "0.2166814498323741" 03:59:25 DEBUG } 03:59:25 DEBUG }, 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG ] 03:59:25 DEBUG "value": [ 03:59:25 DEBUG ] 03:59:25 DEBUG "0" 03:59:25 DEBUG } 03:59:25 DEBUG }, 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG "value": [ 03:59:25 DEBUG "577.4353964912281" 03:59:25 DEBUG } 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG } 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG "value": [ 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG "0" 03:59:25 DEBUG ] 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG 1713063564, 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG } 03:59:25 DEBUG "2" 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG ] 03:59:25 DEBUG } 03:59:25 DEBUG ] 03:59:25 DEBUG } 03:59:25 DEBUG "0" 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG ] 03:59:25 DEBUG ] 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG ] 03:59:25 DEBUG } 03:59:25 DEBUG ] 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG ] 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG ] 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 03:59:25 DEBUG } 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713063625" 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 INFO 04:00:25 INFO [loop_until]: kubectl --namespace=xlou top pods 04:00:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 INFO 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 INFO [loop_until]: kubectl --namespace=xlou top node 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063625" 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713063625" 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063625" 04:00:25 ERROR [http_cmd]: ERROR 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713063625" 04:00:25 ERROR --- status code --- 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063625" 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG --- status code --- 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:00:25 DEBUG --- status code --- 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG --- status code --- 04:00:25 ERROR --- http response --- 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:00:25 DEBUG --- http response --- 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG --- http response --- 04:00:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:00:25 DEBUG { 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG { 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG { 04:00:25 DEBUG { 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:00:25 DEBUG { 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:00:25 DEBUG "data": { 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:00:25 DEBUG "data": { 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG "data": { 04:00:25 DEBUG "data": { 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG { 04:00:25 DEBUG "data": { 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG { 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG { 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG { 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG { 04:00:25 DEBUG "data": { 04:00:25 DEBUG "result": [ 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG "data": { 04:00:25 DEBUG "metric": { 04:00:25 DEBUG { 04:00:25 DEBUG { 04:00:25 DEBUG "data": { 04:00:25 DEBUG "metric": { 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG { 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG "pod": "ds-idrepo-1" 04:00:25 DEBUG "metric": { 04:00:25 DEBUG "metric": { 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG "pod": "ds-idrepo-1" 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "metric": { 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:00:25 DEBUG "result": [ 04:00:25 DEBUG }, 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG "instance": "10.142.0.107:9100" 04:00:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:00:25 DEBUG "result": [ 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG }, 04:00:25 DEBUG { 04:00:25 DEBUG "pod": "ds-idrepo-1" 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG { 04:00:25 DEBUG "value": [ 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG }, 04:00:25 DEBUG }, 04:00:25 DEBUG { 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "metric": { 04:00:25 DEBUG }, 04:00:25 DEBUG { 04:00:25 DEBUG "metric": { 04:00:25 DEBUG 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "metric": { 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG 04:00:25 DEBUG "pod": "ds-idrepo-1" 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG "instance": "10.142.0.107:9100" 04:00:25 DEBUG 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG 04:00:25 DEBUG }, 04:00:25 DEBUG 04:00:25 DEBUG "data": { 04:00:25 DEBUG }, 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:00:25 DEBUG 04:00:25 DEBUG }, 04:00:25 DEBUG 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:00:25 DEBUG "value": [ 04:00:25 DEBUG 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG "value": [ 04:00:25 DEBUG [...] 04:00:25 DEBUG 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG 04:00:25 DEBUG "value": [ 04:00:25 DEBUG [...] 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "result": [ 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG [...] 04:00:25 DEBUG { 04:00:25 DEBUG [...] 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG { 04:00:25 DEBUG 04:00:25 DEBUG [...] 04:00:25 DEBUG { 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "metric": { 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "data": { 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "data": { 04:00:25 DEBUG [...] 04:00:25 DEBUG 04:00:25 DEBUG "pod": "ds-idrepo-1" 04:00:25 DEBUG [...] 04:00:25 DEBUG "pod": "ds-idrepo-0" 04:00:25 DEBUG 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG 04:00:25 DEBUG [...] 04:00:25 DEBUG "pod": "ds-idrepo-0" 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG }, 04:00:25 DEBUG 04:00:25 DEBUG }, 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 DEBUG "instance": "10.142.0.43:9100" 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:00:25 DEBUG }, 04:00:25 DEBUG }, 04:00:25 DEBUG "result": [ 04:00:25 DEBUG 04:00:25 DEBUG "pod": "ds-idrepo-0" 04:00:25 DEBUG "value": [ 04:00:25 DEBUG 04:00:25 DEBUG "value": [ 04:00:25 DEBUG }, 04:00:25 DEBUG { 04:00:25 DEBUG 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "value": [ 04:00:25 DEBUG { 04:00:25 DEBUG 04:00:25 DEBUG }, 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "metric": { 04:00:25 DEBUG 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG "metric": { 04:00:25 DEBUG "pod": "ds-cts-1" 04:00:25 DEBUG "value": [ 04:00:25 DEBUG 04:00:25 DEBUG "instance": "10.142.0.43:9100" 04:00:25 DEBUG "0" 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:00:25 DEBUG 04:00:25 DEBUG "128.8284835423919" 04:00:25 DEBUG "0" 04:00:25 DEBUG "instance": "10.142.0.107:9100" 04:00:25 DEBUG }, 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG 04:00:25 DEBUG }, 04:00:25 DEBUG ] 04:00:25 DEBUG "4416.066666666667" 04:00:25 DEBUG }, 04:00:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:00:25 DEBUG ] 04:00:25 DEBUG ] 04:00:25 DEBUG }, 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "0" 04:00:25 DEBUG [...] 04:00:25 DEBUG "value": [ 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG "value": [ 04:00:25 DEBUG }, 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG "value": [ 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG ] 04:00:25 DEBUG 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG 04:00:25 DEBUG "value": [ 04:00:25 DEBUG ] 04:00:25 DEBUG ] 04:00:25 DEBUG 04:00:25 DEBUG "0" 04:00:25 DEBUG } 04:00:25 DEBUG 04:00:25 DEBUG "0.00033333333333350387" 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG 04:00:25 INFO [loop_until]: OK (rc = 0) 04:00:25 DEBUG --- stdout --- 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 DEBUG ] 04:00:25 DEBUG 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG 04:00:25 DEBUG "330534912" 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG 04:00:25 DEBUG ] 04:00:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5m 8906Mi am-7d597cf7d6-8vr8f 7m 8547Mi am-7d597cf7d6-9wz9h 16m 8471Mi ds-cts-0 193m 7163Mi ds-cts-1 194m 7899Mi ds-cts-2 234m 8111Mi ds-idrepo-0 326m 4584Mi ds-idrepo-1 179m 4301Mi ds-idrepo-2 12m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 997Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 10m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 209m 387Mi 04:00:25 DEBUG } 04:00:25 DEBUG "pod": "ds-idrepo-0" 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG [...] 04:00:25 DEBUG ] 04:00:25 DEBUG [...] 04:00:25 DEBUG } 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 DEBUG --- stderr --- 04:00:25 DEBUG } 04:00:25 DEBUG }, 04:00:25 DEBUG ] 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063625" 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 DEBUG 04:00:25 DEBUG } 04:00:25 DEBUG 04:00:25 DEBUG ] 04:00:25 DEBUG 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 DEBUG "value": [ 04:00:25 DEBUG } 04:00:25 DEBUG 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG ] 04:00:25 DEBUG 04:00:25 DEBUG } 04:00:25 DEBUG 1713063625, 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG } 04:00:25 DEBUG 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG } 04:00:25 DEBUG 04:00:25 DEBUG } 04:00:25 DEBUG "0" 04:00:25 DEBUG --- status code --- 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG } 04:00:25 DEBUG "instance": "10.142.0.43:9100" 04:00:25 DEBUG ] 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG }, 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063625" 04:00:25 DEBUG --- http response --- 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG }, 04:00:25 DEBUG } 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG "value": [ 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063625" 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG "value": [ 04:00:25 DEBUG ] 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG --- status code --- 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG } 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG "0.15547268418485383" 04:00:25 DEBUG ----- output ----- 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG { 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG "0.00033333333333333365" 04:00:25 DEBUG } 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG ] 04:00:25 DEBUG { 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:00:25 DEBUG ] 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG } 04:00:25 DEBUG "status": "success", 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG "data": { 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG } 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG ] 04:00:25 DEBUG "data": { 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:00:25 INFO [http_cmd]: http status code OK 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG { 04:00:25 DEBUG { 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:00:25 DEBUG ] 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:00:25 DEBUG } 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG "result": [ 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG --- status code --- 04:00:25 DEBUG { 04:00:25 DEBUG { 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG } 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG } 04:00:25 DEBUG "result": [ 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG { 04:00:25 DEBUG { 04:00:25 DEBUG { 04:00:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG "data": { 04:00:25 DEBUG "data": { 04:00:25 DEBUG { 04:00:25 DEBUG } 04:00:25 DEBUG { 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:00:25 DEBUG { 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:00:25 DEBUG { 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG "metric": { 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG --- http response --- 04:00:25 DEBUG "data": { 04:00:25 DEBUG "data": { 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG "metric": { 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG "data": { 04:00:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:00:25 DEBUG "data": { 04:00:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG { 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG "data": { 04:00:25 DEBUG "data": { 04:00:25 DEBUG { 04:00:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:00:25 DEBUG { 04:00:25 DEBUG "data": { 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG }, 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG ----- output ----- 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "metric": { 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG }, 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "result": [ 04:00:25 DEBUG { 04:00:25 DEBUG { 04:00:25 DEBUG { 04:00:25 DEBUG "pod": "ds-idrepo-1" 04:00:25 DEBUG { 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "data": { 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "data": { 04:00:25 DEBUG "result": [ 04:00:25 DEBUG { 04:00:25 DEBUG 04:00:25 DEBUG { 04:00:25 DEBUG "status": "success", 04:00:25 DEBUG "metric": { 04:00:25 DEBUG "metric": { 04:00:25 DEBUG }, 04:00:25 DEBUG "metric": { 04:00:25 DEBUG { 04:00:25 DEBUG { 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG { 04:00:25 DEBUG "metric": { 04:00:25 DEBUG 04:00:25 DEBUG "metric": { 04:00:25 DEBUG "data": { 04:00:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:00:25 DEBUG "instance": "10.142.0.107:9100" 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:00:25 DEBUG "metric": { 04:00:25 DEBUG "metric": { 04:00:25 DEBUG "result": [ 04:00:25 DEBUG 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "metric": { 04:00:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:00:25 DEBUG 04:00:25 DEBUG "instance": "10.142.0.107:9100" 04:00:25 DEBUG "resultType": "vector", 04:00:25 DEBUG }, 04:00:25 DEBUG }, 04:00:25 DEBUG 04:00:25 DEBUG }, 04:00:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:00:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:00:25 DEBUG { 04:00:25 DEBUG 04:00:25 DEBUG { 04:00:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:00:25 DEBUG }, 04:00:25 DEBUG [...] 04:00:25 DEBUG }, 04:00:25 DEBUG "result": [ 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "value": [ 04:00:25 DEBUG 04:00:25 DEBUG "value": [ 04:00:25 DEBUG }, 04:00:25 DEBUG }, 04:00:25 DEBUG "metric": { 04:00:25 DEBUG [...] 04:00:25 DEBUG "metric": { 04:00:25 DEBUG }, 04:00:25 DEBUG "value": [ 04:00:25 DEBUG 04:00:25 DEBUG "value": [ 04:00:25 DEBUG { 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "pod": "ds-idrepo-1" 04:00:25 DEBUG 04:00:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:00:25 DEBUG "value": [ 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "metric": { 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG [...] 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG }, 04:00:25 DEBUG 04:00:25 DEBUG }, 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "value": [ 04:00:25 DEBUG 04:00:25 DEBUG "value": [ 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:00:25 DEBUG 04:00:25 DEBUG }, 04:00:25 DEBUG [...] 04:00:25 DEBUG [...] 04:00:25 DEBUG 04:00:25 DEBUG [...] 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG [...] 04:00:25 DEBUG }, 04:00:25 DEBUG [...] 04:00:25 DEBUG "value": [ 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG [...] 04:00:25 DEBUG [...] 04:00:25 DEBUG 04:00:25 DEBUG }, 04:00:25 DEBUG 04:00:25 DEBUG [...] 04:00:25 DEBUG 04:00:25 DEBUG "value": [ 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "pod": "ds-cts-1" 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "value": [ 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG }, 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG [...] 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG [...] 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "0" 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:00:25 DEBUG "instance": "10.142.0.43:9100" 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "5092074.640496582" 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:00:25 DEBUG ] 04:00:25 DEBUG "instance": "10.142.0.43:9100" 04:00:25 DEBUG [...] 04:00:25 DEBUG }, 04:00:25 DEBUG }, 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG }, 04:00:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:00:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:00:25 DEBUG }, 04:00:25 DEBUG ] 04:00:25 DEBUG 04:00:25 DEBUG 04:00:25 DEBUG }, 04:00:25 DEBUG } 04:00:25 DEBUG }, 04:00:25 DEBUG 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "0" 04:00:25 DEBUG "value": [ 04:00:25 DEBUG }, 04:00:25 DEBUG "value": [ 04:00:25 DEBUG } 04:00:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:00:25 DEBUG "value": [ 04:00:25 DEBUG ] 04:00:25 DEBUG "value": [ 04:00:25 DEBUG 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG "0.015" 04:00:25 DEBUG ] 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG "value": [ 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG ] 04:00:25 DEBUG }, 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG "0" 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG "0.1930603876552286" 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG "0" 04:00:25 DEBUG } 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:00:25 DEBUG 04:00:25 DEBUG "value": [ 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG "12685224960" 04:00:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG ] 04:00:25 DEBUG "0" 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG "5678.633333333333" 04:00:25 DEBUG }, 04:00:25 DEBUG 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG }, 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG "value": [ 04:00:25 DEBUG "pod": "ds-idrepo-0" 04:00:25 DEBUG "128.8284835423919" 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG }, 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG "value": [ 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG "2" 04:00:25 DEBUG "value": [ 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG "0" 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG 1713063625, 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG "0" 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG ] 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 DEBUG } 04:00:25 INFO [loop_until]: OK (rc = 0) 04:00:25 DEBUG --- stdout --- 04:00:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4819Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 255m 0% 10234Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 52m 0% 10963Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 89m 0% 3066Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 150m 0% 6341Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 358m 1% 10606Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2911Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 56m 0% 9977Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 184m 0% 5456Mi 4% gke-xlou-cdm-ds-0f601751-j488 253m 0% 9342Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2214Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 92m 0% 10005Mi 8% gke-xlou-cdm-ds-0f601751-nmxm 154m 0% 5995Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 401m 2% 4182Mi 7% 04:00:25 DEBUG --- stderr --- 04:00:25 DEBUG 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713063685" 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 ERROR [http_cmd]: ERROR 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 ERROR --- status code --- 04:01:25 DEBUG --- status code --- 04:01:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 ERROR --- http response --- 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:01:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:01:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG { 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG "data": { 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "result": [ 04:01:25 DEBUG { 04:01:25 DEBUG "metric": { 04:01:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:01:25 DEBUG }, 04:01:25 DEBUG "value": [ 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG [...] 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:01:25 DEBUG }, 04:01:25 DEBUG "value": [ 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG "0" 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063685" 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG --- http response --- 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 INFO 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063685" 04:01:25 INFO [loop_until]: kubectl --namespace=xlou top pods 04:01:25 DEBUG ----- output ----- 04:01:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:01:25 DEBUG { 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG "data": { 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "result": [ 04:01:25 DEBUG { 04:01:25 DEBUG "metric": { 04:01:25 DEBUG "pod": "ds-idrepo-1" 04:01:25 DEBUG }, 04:01:25 DEBUG "value": [ 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG [...] 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG "pod": "ds-idrepo-0" 04:01:25 DEBUG }, 04:01:25 DEBUG "value": [ 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG "0" 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713063685" 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:01:25 DEBUG ----- output ----- 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG { 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG --- status code --- 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG "data": { 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG --- http response --- 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG "result": [ 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG { 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG { 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG "metric": { 04:01:25 DEBUG { 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG "instance": "10.142.0.107:9100" 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG "data": { 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG { 04:01:25 DEBUG }, 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG "data": { 04:01:25 DEBUG { 04:01:25 DEBUG { 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG { 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG "value": [ 04:01:25 DEBUG { 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG "result": [ 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG "data": { 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 DEBUG 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "data": { 04:01:25 DEBUG "data": { 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG "data": { 04:01:25 DEBUG { 04:01:25 DEBUG "data": { 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG "result": [ 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "metric": { 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "result": [ 04:01:25 DEBUG 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG { 04:01:25 DEBUG "result": [ 04:01:25 DEBUG "result": [ 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG "result": [ 04:01:25 DEBUG "pod": "ds-idrepo-1" 04:01:25 DEBUG "result": [ 04:01:25 DEBUG { 04:01:25 DEBUG [...] 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG "metric": { 04:01:25 DEBUG { 04:01:25 DEBUG { 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG { 04:01:25 DEBUG }, 04:01:25 DEBUG { 04:01:25 DEBUG "metric": { 04:01:25 DEBUG 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG "pod": "ds-idrepo-1" 04:01:25 DEBUG }, 04:01:25 DEBUG "metric": { 04:01:25 DEBUG "metric": { 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "metric": { 04:01:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:01:25 DEBUG 04:01:25 DEBUG "metric": { 04:01:25 DEBUG "value": [ 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:01:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713063685" 04:01:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG 04:01:25 DEBUG "pod": "ds-idrepo-1" 04:01:25 DEBUG }, 04:01:25 DEBUG 04:01:25 DEBUG "instance": "10.142.0.107:9100" 04:01:25 DEBUG 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG }, 04:01:25 DEBUG "value": [ 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063685" 04:01:25 DEBUG { 04:01:25 DEBUG 04:01:25 DEBUG }, 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "instance": "10.142.0.43:9100" 04:01:25 DEBUG }, 04:01:25 DEBUG 04:01:25 DEBUG { 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG "status": "success", 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG 04:01:25 DEBUG "value": [ 04:01:25 DEBUG 04:01:25 DEBUG }, 04:01:25 DEBUG "value": [ 04:01:25 DEBUG 04:01:25 DEBUG }, 04:01:25 DEBUG "data": { 04:01:25 DEBUG 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG "data": { 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG [...] 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG "value": [ 04:01:25 DEBUG 04:01:25 DEBUG [...] 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG "result": [ 04:01:25 DEBUG [...] 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG "result": [ 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG [...] 04:01:25 DEBUG "2833.6666666666665" 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG { 04:01:25 DEBUG 04:01:25 DEBUG { 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:01:25 DEBUG 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:01:25 DEBUG [...] 04:01:25 DEBUG 04:01:25 DEBUG ] 04:01:25 DEBUG [...] 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG "metric": { 04:01:25 DEBUG 04:01:25 DEBUG "metric": { 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG "pod": "ds-idrepo-0" 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG } 04:01:25 DEBUG "pod": "ds-idrepo-0" 04:01:25 DEBUG [...] 04:01:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:01:25 DEBUG 04:01:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:01:25 DEBUG { 04:01:25 DEBUG }, 04:01:25 DEBUG { 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:01:25 DEBUG }, 04:01:25 DEBUG }, 04:01:25 DEBUG 04:01:25 DEBUG }, 04:01:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:01:25 DEBUG }, 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG "value": [ 04:01:25 DEBUG 04:01:25 DEBUG "value": [ 04:01:25 DEBUG }, 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "data": { 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG "data": { 04:01:25 DEBUG "pod": "ds-cts-1" 04:01:25 DEBUG 04:01:25 DEBUG "value": [ 04:01:25 DEBUG } 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG "value": [ 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "0" 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG }, 04:01:25 DEBUG "instance": "10.142.0.43:9100" 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG "0" 04:01:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG "0" 04:01:25 DEBUG "result": [ 04:01:25 DEBUG ] 04:01:25 DEBUG "result": [ 04:01:25 DEBUG "value": [ 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG --- status code --- 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG "0.015" 04:01:25 DEBUG ] 04:01:25 DEBUG }, 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG ] 04:01:25 DEBUG { 04:01:25 DEBUG } 04:01:25 DEBUG { 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG }, 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG "value": [ 04:01:25 DEBUG [...] 04:01:25 DEBUG 04:01:25 DEBUG } 04:01:25 DEBUG "metric": { 04:01:25 DEBUG ] 04:01:25 DEBUG "metric": { 04:01:25 DEBUG "0" 04:01:25 DEBUG "value": [ 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG } 04:01:25 DEBUG ] 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG 04:01:25 DEBUG [...] 04:01:25 DEBUG ] 04:01:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:01:25 DEBUG } 04:01:25 DEBUG "pod": "ds-idrepo-1" 04:01:25 DEBUG ] 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG "0" 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG } 04:01:25 DEBUG }, 04:01:25 DEBUG } 04:01:25 DEBUG }, 04:01:25 DEBUG } 04:01:25 DEBUG ] 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG ] 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG } 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "0.00020000000000000757" 04:01:25 DEBUG } 04:01:25 DEBUG { 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG { 04:01:25 DEBUG ] 04:01:25 DEBUG }, 04:01:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG } 04:01:25 DEBUG "data": { 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG } 04:01:25 DEBUG "value": [ 04:01:25 DEBUG }, 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG ] 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "data": { 04:01:25 DEBUG } 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG "value": [ 04:01:25 DEBUG [...] 04:01:25 DEBUG [...] 04:01:25 DEBUG } 04:01:25 DEBUG "result": [ 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "331026432" 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 DEBUG } 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 DEBUG { 04:01:25 DEBUG "result": [ 04:01:25 DEBUG ] 04:01:25 DEBUG "29819.54854499433" 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 DEBUG "metric": { 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063685" 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063685" 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 DEBUG { 04:01:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063685" 04:01:25 DEBUG } 04:01:25 DEBUG ] 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:01:25 DEBUG "metric": { 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:01:25 INFO [loop_until]: OK (rc = 0) 04:01:25 DEBUG "pod": "ds-idrepo-0" 04:01:25 DEBUG }, 04:01:25 DEBUG "instance": "10.142.0.107:9100" 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG } 04:01:25 DEBUG ] 04:01:25 DEBUG }, 04:01:25 DEBUG --- stdout --- 04:01:25 DEBUG }, 04:01:25 DEBUG "value": [ 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG }, 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "value": [ 04:01:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8906Mi am-7d597cf7d6-8vr8f 7m 8546Mi am-7d597cf7d6-9wz9h 12m 8471Mi ds-cts-0 168m 7163Mi ds-cts-1 185m 7900Mi ds-cts-2 197m 8111Mi ds-idrepo-0 11m 4582Mi ds-idrepo-1 11m 4301Mi ds-idrepo-2 11m 4104Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 996Mi idm-b85f9d978-rl76f 5m 988Mi lodemon-84dfb5c75c-qnqlj 10m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 223Mi 04:01:25 DEBUG 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG "value": [ 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG } 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG 1713063685, 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 INFO [http_cmd]: http status code OK 04:01:25 DEBUG --- stderr --- 04:01:25 DEBUG 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:01:25 DEBUG "0" 04:01:25 DEBUG "0" 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG --- status code --- 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:01:25 DEBUG ] 04:01:25 DEBUG ] 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:01:25 DEBUG [...] 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:01:25 DEBUG 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:01:25 DEBUG { 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG --- http response --- 04:01:25 DEBUG 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG [...] 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG { 04:01:25 DEBUG ] 04:01:25 DEBUG ] 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:01:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:01:25 DEBUG 04:01:25 DEBUG { 04:01:25 DEBUG { 04:01:25 DEBUG 04:01:25 DEBUG { 04:01:25 DEBUG { 04:01:25 DEBUG "data": { 04:01:25 DEBUG { 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG ----- output ----- 04:01:25 DEBUG 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG "data": { 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG { 04:01:25 DEBUG { 04:01:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:01:25 DEBUG "data": { 04:01:25 DEBUG "data": { 04:01:25 DEBUG 04:01:25 DEBUG "data": { 04:01:25 DEBUG "data": { 04:01:25 DEBUG "result": [ 04:01:25 DEBUG "data": { 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG "status": "success", 04:01:25 DEBUG }, 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "instance": "10.142.0.43:9100" 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG { 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "result": [ 04:01:25 DEBUG "data": { 04:01:25 DEBUG "data": { 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "result": [ 04:01:25 DEBUG "result": [ 04:01:25 DEBUG }, 04:01:25 DEBUG "result": [ 04:01:25 DEBUG "result": [ 04:01:25 DEBUG "metric": { 04:01:25 DEBUG "result": [ 04:01:25 DEBUG { 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG "resultType": "vector", 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG { 04:01:25 DEBUG { 04:01:25 DEBUG "value": [ 04:01:25 DEBUG { 04:01:25 DEBUG { 04:01:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:01:25 DEBUG { 04:01:25 DEBUG "metric": { 04:01:25 DEBUG "result": [ 04:01:25 DEBUG "result": [ 04:01:25 DEBUG "0.034612828715485454" 04:01:25 DEBUG "metric": { 04:01:25 DEBUG "metric": { 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG "metric": { 04:01:25 DEBUG "metric": { 04:01:25 DEBUG }, 04:01:25 DEBUG "metric": { 04:01:25 DEBUG "pod": "ds-idrepo-1" 04:01:25 DEBUG { 04:01:25 DEBUG { 04:01:25 DEBUG ] 04:01:25 DEBUG "instance": "10.142.0.107:9100" 04:01:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:01:25 DEBUG "0" 04:01:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:01:25 DEBUG "instance": "10.142.0.107:9100" 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:01:25 DEBUG }, 04:01:25 DEBUG "metric": { 04:01:25 DEBUG "metric": { 04:01:25 DEBUG } 04:01:25 DEBUG }, 04:01:25 DEBUG }, 04:01:25 DEBUG ] 04:01:25 DEBUG }, 04:01:25 DEBUG }, 04:01:25 DEBUG 04:01:25 DEBUG }, 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "pod": "ds-idrepo-1" 04:01:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:01:25 DEBUG ] 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "value": [ 04:01:25 DEBUG } 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "value": [ 04:01:25 DEBUG 04:01:25 DEBUG "value": [ 04:01:25 DEBUG 04:01:25 DEBUG }, 04:01:25 DEBUG }, 04:01:25 DEBUG } 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG ] 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "value": [ 04:01:25 DEBUG } 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG } 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG [...] 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG } 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG [...] 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG [...] 04:01:25 DEBUG [...] 04:01:25 DEBUG [...] 04:01:25 DEBUG [...] 04:01:25 DEBUG 04:01:25 DEBUG [...] 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG [...] 04:01:25 DEBUG [...] 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG }, 04:01:25 DEBUG 04:01:25 DEBUG "pod": "ds-cts-1" 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG "instance": "10.142.0.43:9100" 04:01:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:01:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:01:25 DEBUG "instance": "10.142.0.43:9100" 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:01:25 DEBUG }, 04:01:25 DEBUG 04:01:25 DEBUG 04:01:25 DEBUG }, 04:01:25 DEBUG }, 04:01:25 DEBUG }, 04:01:25 DEBUG }, 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG }, 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "pod": "ds-idrepo-0" 04:01:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "0" 04:01:25 DEBUG "value": [ 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG }, 04:01:25 DEBUG "value": [ 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG ] 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG "0" 04:01:25 DEBUG }, 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG "12686056448" 04:01:25 DEBUG "0" 04:01:25 DEBUG "0" 04:01:25 DEBUG "5163.233333333334" 04:01:25 DEBUG } 04:01:25 DEBUG "0.005264999623834216" 04:01:25 DEBUG ] 04:01:25 DEBUG "value": [ 04:01:25 DEBUG "0" 04:01:25 DEBUG ] 04:01:25 DEBUG ] 04:01:25 DEBUG ] 04:01:25 DEBUG ] 04:01:25 DEBUG ] 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG 1713063685, 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG ] 04:01:25 DEBUG "2" 04:01:25 DEBUG } 04:01:25 DEBUG ] 04:01:25 DEBUG ] 04:01:25 DEBUG ] 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG ] 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG ] 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 DEBUG } 04:01:25 INFO 04:01:25 INFO [loop_until]: kubectl --namespace=xlou top node 04:01:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:01:25 INFO [loop_until]: OK (rc = 0) 04:01:25 DEBUG --- stdout --- 04:01:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4819Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 258m 0% 10239Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 50m 0% 10960Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 90m 0% 3063Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 65m 0% 6343Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 296m 0% 10597Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2924Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 61m 0% 9967Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 5471Mi 4% gke-xlou-cdm-ds-0f601751-j488 249m 0% 9350Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2208Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 97m 0% 10012Mi 8% gke-xlou-cdm-ds-0f601751-nmxm 61m 0% 6022Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 68m 0% 4030Mi 6% 04:01:25 DEBUG --- stderr --- 04:01:25 DEBUG 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713063745" 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 ERROR [http_cmd]: ERROR 04:02:25 ERROR --- status code --- 04:02:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:02:25 ERROR --- http response --- 04:02:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:02:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "data": { 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "pod": "ds-idrepo-1" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG [...] 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "pod": "ds-idrepo-0" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG "0" 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063745" 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "data": { 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG [...] 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG "0" 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- http response --- 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "data": { 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG [...] 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG "0.015" 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "data": { 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG [...] 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG "0" 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "data": { 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "instance": "10.142.0.107:9100" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG [...] 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "instance": "10.142.0.43:9100" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG "2855.733333333333" 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "data": { 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "pod": "ds-idrepo-1" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG [...] 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "pod": "ds-idrepo-0" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG "0" 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063745" 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "data": { 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "instance": "10.142.0.107:9100" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG [...] 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "instance": "10.142.0.43:9100" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG "0.00030000000000001136" 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 INFO 04:02:25 INFO [loop_until]: kubectl --namespace=xlou top pods 04:02:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063745" 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG --- status code --- 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:02:25 DEBUG --- http response --- 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG { 04:02:25 DEBUG { 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG { 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "data": { 04:02:25 DEBUG "data": { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "data": { 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "resultType": "vector", 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG "data": { 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG "result": [ 04:02:25 DEBUG "result": [ 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG { 04:02:25 DEBUG { 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "data": { 04:02:25 DEBUG "pod": "ds-idrepo-1" 04:02:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG }, 04:02:25 DEBUG }, 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG "pod": "ds-idrepo-1" 04:02:25 DEBUG }, 04:02:25 DEBUG "result": [ 04:02:25 DEBUG "value": [ 04:02:25 DEBUG "value": [ 04:02:25 DEBUG { 04:02:25 DEBUG { 04:02:25 DEBUG { 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG { 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG "metric": { 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "data": { 04:02:25 DEBUG "data": { 04:02:25 DEBUG "data": { 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG }, 04:02:25 DEBUG [...] 04:02:25 DEBUG [...] 04:02:25 DEBUG "result": [ 04:02:25 DEBUG "result": [ 04:02:25 DEBUG "result": [ 04:02:25 DEBUG 04:02:25 DEBUG [...] 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG { 04:02:25 DEBUG { 04:02:25 DEBUG { 04:02:25 DEBUG [...] 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "pod": "ds-idrepo-1" 04:02:25 DEBUG "instance": "10.142.0.107:9100" 04:02:25 DEBUG "pod": "ds-idrepo-1" 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "pod": "ds-cts-1" 04:02:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:02:25 DEBUG }, 04:02:25 DEBUG }, 04:02:25 DEBUG }, 04:02:25 DEBUG 04:02:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:02:25 DEBUG [...] 04:02:25 DEBUG }, 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG "value": [ 04:02:25 DEBUG "value": [ 04:02:25 DEBUG "pod": "ds-idrepo-0" 04:02:25 DEBUG }, 04:02:25 DEBUG 04:02:25 DEBUG "value": [ 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713063745" 04:02:25 DEBUG 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 1713063745, 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 DEBUG 04:02:25 DEBUG "0" 04:02:25 DEBUG "0" 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG "0" 04:02:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:02:25 DEBUG ] 04:02:25 DEBUG ] 04:02:25 DEBUG [...] 04:02:25 DEBUG [...] 04:02:25 DEBUG [...] 04:02:25 DEBUG "0" 04:02:25 DEBUG ] 04:02:25 DEBUG }, 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG "value": [ 04:02:25 DEBUG ] 04:02:25 DEBUG ] 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG "0" 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG "pod": "ds-idrepo-0" 04:02:25 DEBUG "instance": "10.142.0.43:9100" 04:02:25 DEBUG "pod": "ds-cts-1" 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG }, 04:02:25 DEBUG }, 04:02:25 DEBUG }, 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG "value": [ 04:02:25 DEBUG "value": [ 04:02:25 DEBUG "value": [ 04:02:25 DEBUG ] 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG } 04:02:25 DEBUG "0" 04:02:25 DEBUG "12686578688" 04:02:25 DEBUG "0" 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG ] 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG ] 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "data": { 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG [...] 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG "0" 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG --- status code --- 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713063745" 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- http response --- 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063745" 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "data": { 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG [...] 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG "330477568" 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "data": { 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "pod": "ds-idrepo-1" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG [...] 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "pod": "ds-idrepo-0" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG "0" 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063745" 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063745" 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "data": { 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG [...] 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG "0.026909816736767445" 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG { 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG "data": { 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "instance": "10.142.0.107:9100" 04:02:25 DEBUG }, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:02:25 DEBUG 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG 04:02:25 DEBUG { 04:02:25 DEBUG [...] 04:02:25 DEBUG "status": "success", 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG 04:02:25 DEBUG "data": { 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG 04:02:25 DEBUG "result": [ 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG "instance": "10.142.0.43:9100" 04:02:25 DEBUG { 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:02:25 DEBUG }, 04:02:25 DEBUG "metric": { 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG "value": [ 04:02:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG { 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG }, 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "0.0003333333333333262" 04:02:25 DEBUG "value": [ 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:02:25 DEBUG "data": { 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:02:25 INFO [http_cmd]: http status code OK 04:02:25 DEBUG ] 04:02:25 DEBUG 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG --- status code --- 04:02:25 DEBUG } 04:02:25 DEBUG 04:02:25 DEBUG { 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:02:25 DEBUG ] 04:02:25 DEBUG 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG --- http response --- 04:02:25 DEBUG } 04:02:25 DEBUG [...] 04:02:25 DEBUG "data": { 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "data": { 04:02:25 DEBUG } 04:02:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:02:25 DEBUG 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG ----- output ----- 04:02:25 DEBUG 04:02:25 DEBUG "result": [ 04:02:25 DEBUG }, 04:02:25 DEBUG "result": [ 04:02:25 DEBUG { 04:02:25 DEBUG 04:02:25 DEBUG { 04:02:25 DEBUG "value": [ 04:02:25 DEBUG { 04:02:25 DEBUG "status": "success", 04:02:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:02:25 DEBUG "metric": { 04:02:25 DEBUG 04:02:25 DEBUG "metric": { 04:02:25 DEBUG "data": { 04:02:25 DEBUG }, 04:02:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:02:25 DEBUG 04:02:25 DEBUG "instance": "10.142.0.107:9100" 04:02:25 DEBUG "resultType": "vector", 04:02:25 DEBUG "value": [ 04:02:25 DEBUG }, 04:02:25 DEBUG 04:02:25 DEBUG }, 04:02:25 DEBUG "result": [ 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG "value": [ 04:02:25 DEBUG [...] 04:02:25 DEBUG "value": [ 04:02:25 DEBUG { 04:02:25 DEBUG "0" 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "metric": { 04:02:25 DEBUG ] 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:02:25 DEBUG } 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG }, 04:02:25 DEBUG ] 04:02:25 DEBUG [...] 04:02:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:02:25 DEBUG [...] 04:02:25 DEBUG "value": [ 04:02:25 DEBUG } 04:02:25 DEBUG 04:02:25 DEBUG }, 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG } 04:02:25 DEBUG 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG 04:02:25 DEBUG 04:02:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:02:25 DEBUG "0.000143021636021605" 04:02:25 DEBUG "instance": "10.142.0.43:9100" 04:02:25 DEBUG [...] 04:02:25 DEBUG }, 04:02:25 DEBUG ] 04:02:25 DEBUG }, 04:02:25 DEBUG 04:02:25 DEBUG "value": [ 04:02:25 DEBUG } 04:02:25 DEBUG "value": [ 04:02:25 DEBUG 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG ] 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG 04:02:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:02:25 DEBUG } 04:02:25 DEBUG "5197.266666666667" 04:02:25 DEBUG "0" 04:02:25 INFO [loop_until]: OK (rc = 0) 04:02:25 DEBUG }, 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG ] 04:02:25 DEBUG --- stdout --- 04:02:25 DEBUG "value": [ 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5m 8906Mi am-7d597cf7d6-8vr8f 7m 8546Mi am-7d597cf7d6-9wz9h 18m 8471Mi ds-cts-0 433m 7163Mi ds-cts-1 268m 7900Mi ds-cts-2 173m 8111Mi ds-idrepo-0 11m 4582Mi ds-idrepo-1 11m 4300Mi ds-idrepo-2 10m 4104Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 14m 997Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 9m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 223Mi 04:02:25 DEBUG 1713063745, 04:02:25 DEBUG ] 04:02:25 DEBUG ] 04:02:25 DEBUG --- stderr --- 04:02:25 DEBUG "0" 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 DEBUG ] 04:02:25 DEBUG } 04:02:25 DEBUG } 04:02:25 INFO 04:02:25 INFO [loop_until]: kubectl --namespace=xlou top node 04:02:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:02:25 INFO [loop_until]: OK (rc = 0) 04:02:25 DEBUG --- stdout --- 04:02:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 4819Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 439m 1% 10212Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 54m 0% 10965Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 86m 0% 3073Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 65m 0% 6338Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 518m 1% 10591Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 66m 0% 2923Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 59m 0% 9976Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 62m 0% 5474Mi 4% gke-xlou-cdm-ds-0f601751-j488 414m 1% 9464Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2208Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 92m 0% 10010Mi 8% gke-xlou-cdm-ds-0f601751-nmxm 61m 0% 6032Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 66m 0% 4029Mi 6% 04:02:25 DEBUG --- stderr --- 04:02:25 DEBUG 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 ERROR [http_cmd]: ERROR 04:03:25 ERROR --- status code --- 04:03:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:03:25 ERROR --- http response --- 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG --- status code --- 04:03:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 04:03:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG { 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "data": { 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "result": [ 04:03:25 DEBUG { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "pod": "ds-cts-0" 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG [...] 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG "0" 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 DEBUG --- status code --- 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG { 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG { 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "data": { 04:03:25 DEBUG "data": { 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "data": { 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "data": { 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "result": [ 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG "result": [ 04:03:25 DEBUG { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:03:25 DEBUG "pod": "ds-idrepo-1" 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "instance": "10.142.0.107:9100" 04:03:25 DEBUG }, 04:03:25 DEBUG }, 04:03:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "value": [ 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG [...] 04:03:25 DEBUG [...] 04:03:25 DEBUG 04:03:25 DEBUG [...] 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG [...] 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:03:25 DEBUG "pod": "ds-idrepo-0" 04:03:25 DEBUG 04:03:25 DEBUG "instance": "10.142.0.43:9100" 04:03:25 DEBUG }, 04:03:25 DEBUG }, 04:03:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "value": [ 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG "0.015" 04:03:25 DEBUG "0" 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG "2837.2" 04:03:25 DEBUG ] 04:03:25 DEBUG ] 04:03:25 DEBUG "1995.6" 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG { 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "data": { 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "result": [ 04:03:25 DEBUG { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "pod": "ds-idrepo-1" 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG [...] 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG "pod": "ds-idrepo-0" 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG "0" 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063805" 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG { 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "data": { 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "result": [ 04:03:25 DEBUG { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "pod": "ds-idrepo-1" 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG [...] 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG "pod": "ds-idrepo-0" 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG "0" 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG { 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "data": { 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "result": [ 04:03:25 DEBUG { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "instance": "10.142.0.107:9100" 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG [...] 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG "instance": "10.142.0.43:9100" 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG "0.0004333333333332708" 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713063805" 04:03:25 INFO 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [loop_until]: kubectl --namespace=xlou top pods 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 DEBUG --- status code --- 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:03:25 DEBUG --- http response --- 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG { 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG "data": { 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "data": { 04:03:25 DEBUG "data": { 04:03:25 DEBUG "data": { 04:03:25 DEBUG "data": { 04:03:25 DEBUG "data": { 04:03:25 DEBUG "data": { 04:03:25 DEBUG "data": { 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "data": { 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG "data": { 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG { 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "result": [ 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "result": [ 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG { 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "metric": { 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG { 04:03:25 DEBUG "data": { 04:03:25 DEBUG { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "data": { 04:03:25 DEBUG "pod": "ds-idrepo-1" 04:03:25 DEBUG { 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "pod": "ds-idrepo-1" 04:03:25 DEBUG "instance": "10.142.0.107:9100" 04:03:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:03:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:03:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:03:25 DEBUG "pod": "ds-idrepo-1" 04:03:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG }, 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:03:25 DEBUG }, 04:03:25 DEBUG }, 04:03:25 DEBUG }, 04:03:25 DEBUG }, 04:03:25 DEBUG }, 04:03:25 DEBUG }, 04:03:25 DEBUG }, 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "data": { 04:03:25 DEBUG }, 04:03:25 DEBUG { 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "value": [ 04:03:25 DEBUG { 04:03:25 DEBUG 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063805" 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG "metric": { 04:03:25 DEBUG 04:03:25 DEBUG "result": [ 04:03:25 DEBUG 04:03:25 DEBUG "instance": "10.142.0.107:9100" 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG "pod": "ds-idrepo-1" 04:03:25 DEBUG 04:03:25 DEBUG { 04:03:25 DEBUG 04:03:25 DEBUG }, 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG }, 04:03:25 DEBUG [...] 04:03:25 DEBUG "metric": { 04:03:25 DEBUG 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG [...] 04:03:25 DEBUG [...] 04:03:25 DEBUG [...] 04:03:25 DEBUG [...] 04:03:25 DEBUG [...] 04:03:25 DEBUG [...] 04:03:25 DEBUG [...] 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG "pod": "ds-cts-2" 04:03:25 DEBUG [...] 04:03:25 DEBUG 04:03:25 DEBUG [...] 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG }, 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063805" 04:03:25 DEBUG 04:03:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG "pod": "ds-cts-1" 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG [...] 04:03:25 DEBUG 04:03:25 DEBUG "pod": "ds-idrepo-0" 04:03:25 DEBUG "instance": "10.142.0.43:9100" 04:03:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:03:25 DEBUG }, 04:03:25 DEBUG 04:03:25 DEBUG "pod": "ds-idrepo-0" 04:03:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:03:25 DEBUG [...] 04:03:25 DEBUG }, 04:03:25 DEBUG 04:03:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG }, 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:03:25 DEBUG }, 04:03:25 DEBUG }, 04:03:25 DEBUG 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG }, 04:03:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:03:25 DEBUG }, 04:03:25 DEBUG "instance": "10.142.0.43:9100" 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG [...] 04:03:25 DEBUG "value": [ 04:03:25 DEBUG }, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG "813068288" 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG 04:03:25 DEBUG "0.000024874999999999998" 04:03:25 DEBUG 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG }, 04:03:25 DEBUG "0" 04:03:25 DEBUG "5070.233333333334" 04:03:25 DEBUG "2.7773966942638366" 04:03:25 DEBUG ] 04:03:25 DEBUG "0" 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG "0" 04:03:25 DEBUG "pod": "ds-cts-1" 04:03:25 DEBUG ] 04:03:25 DEBUG 04:03:25 DEBUG "1990.9666666666667" 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG "value": [ 04:03:25 DEBUG ] 04:03:25 DEBUG ] 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG "0" 04:03:25 DEBUG ] 04:03:25 DEBUG }, 04:03:25 DEBUG } 04:03:25 DEBUG 04:03:25 DEBUG ] 04:03:25 DEBUG "1991.6" 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG "value": [ 04:03:25 DEBUG ] 04:03:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG "12687193088" 04:03:25 DEBUG ] 04:03:25 DEBUG ] 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG --- http response --- 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG "0" 04:03:25 INFO [http_cmd]: http status code OK 04:03:25 DEBUG }, 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG ] 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG --- status code --- 04:03:25 DEBUG "value": [ 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG } 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG { 04:03:25 DEBUG --- http response --- 04:03:25 DEBUG } 04:03:25 DEBUG { 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 04:03:25 DEBUG "0" 04:03:25 DEBUG } 04:03:25 INFO [loop_until]: OK (rc = 0) 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG } 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG ] 04:03:25 DEBUG --- stdout --- 04:03:25 DEBUG } 04:03:25 DEBUG "data": { 04:03:25 DEBUG ----- output ----- 04:03:25 DEBUG "data": { 04:03:25 DEBUG } 04:03:25 DEBUG { 04:03:25 DEBUG } 04:03:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 450m 8911Mi am-7d597cf7d6-8vr8f 7m 8546Mi am-7d597cf7d6-9wz9h 238m 8485Mi ds-cts-0 573m 7170Mi ds-cts-1 537m 7906Mi ds-cts-2 186m 8114Mi ds-idrepo-0 1801m 4584Mi ds-idrepo-1 147m 4301Mi ds-idrepo-2 26m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 997Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 8m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 2714m 685Mi 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG { 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG ] 04:03:25 DEBUG --- stderr --- 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "status": "success", 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "data": { 04:03:25 DEBUG } 04:03:25 DEBUG 04:03:25 DEBUG { 04:03:25 DEBUG "data": { 04:03:25 DEBUG { 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG } 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "resultType": "vector", 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "pod": "ds-cts-2" 04:03:25 DEBUG "result": [ 04:03:25 DEBUG "instance": "10.142.0.107:9100" 04:03:25 DEBUG { 04:03:25 DEBUG }, 04:03:25 DEBUG { 04:03:25 DEBUG }, 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "metric": { 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "pod": "ds-cts-2" 04:03:25 DEBUG 04:03:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:03:25 DEBUG 04:03:25 DEBUG }, 04:03:25 DEBUG 04:03:25 DEBUG }, 04:03:25 DEBUG 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG [...] 04:03:25 DEBUG 04:03:25 DEBUG [...] 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG [...] 04:03:25 DEBUG 04:03:25 DEBUG [...] 04:03:25 DEBUG 04:03:25 DEBUG 04:03:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:03:25 DEBUG 04:03:25 DEBUG "instance": "10.142.0.43:9100" 04:03:25 DEBUG 04:03:25 DEBUG }, 04:03:25 DEBUG 04:03:25 DEBUG }, 04:03:25 DEBUG 04:03:25 DEBUG "value": [ 04:03:25 DEBUG 04:03:25 DEBUG "value": [ 04:03:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG }, 04:03:25 DEBUG "0" 04:03:25 DEBUG }, 04:03:25 DEBUG "0.00033333333333333365" 04:03:25 DEBUG "value": [ 04:03:25 DEBUG ] 04:03:25 DEBUG "value": [ 04:03:25 DEBUG ] 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG } 04:03:25 DEBUG 1713063805, 04:03:25 DEBUG } 04:03:25 DEBUG "0" 04:03:25 DEBUG ] 04:03:25 DEBUG "33230.5" 04:03:25 DEBUG ] 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG ] 04:03:25 DEBUG ] 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 DEBUG } 04:03:25 INFO 04:03:25 INFO [loop_until]: kubectl --namespace=xlou top node 04:03:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:03:25 INFO [loop_until]: OK (rc = 0) 04:03:25 DEBUG --- stdout --- 04:03:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4830Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 279m 0% 10241Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1270m 4% 10972Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 1886m 6% 3041Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 284m 0% 6364Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 426m 1% 10616Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2923Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1324m 4% 9992Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 189m 0% 5493Mi 4% gke-xlou-cdm-ds-0f601751-j488 790m 2% 9365Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2218Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 661m 2% 10039Mi 8% gke-xlou-cdm-ds-0f601751-nmxm 1981m 6% 6078Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 2676m 16% 4470Mi 7% 04:03:25 DEBUG --- stderr --- 04:03:25 DEBUG 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG { 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "data": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "result": [ 04:04:25 DEBUG { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "pod": "ds-idrepo-1" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "pod": "ds-idrepo-0" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "0" 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG { 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "data": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "result": [ 04:04:25 DEBUG { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "pod": "idm-b85f9d978-j5pbm" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "0" 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG { 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "data": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "result": [ 04:04:25 DEBUG { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "0.015" 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG { 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "data": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "result": [ 04:04:25 DEBUG { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "instance": "10.142.0.107:9100" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "instance": "10.142.0.43:9100" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "2891.7" 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 ERROR [http_cmd]: ERROR 04:04:25 ERROR --- status code --- 04:04:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:04:25 ERROR --- http response --- 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG --- status code --- 04:04:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG { 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "data": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "result": [ 04:04:25 DEBUG { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "2677.766666666667" 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063865" 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG { 04:04:25 DEBUG { 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "data": { 04:04:25 DEBUG "data": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "result": [ 04:04:25 DEBUG "result": [ 04:04:25 DEBUG { 04:04:25 DEBUG { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "pod": "ds-idrepo-1" 04:04:25 DEBUG "pod": "ds-idrepo-1" 04:04:25 DEBUG }, 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "pod": "ds-idrepo-0" 04:04:25 DEBUG "pod": "ds-idrepo-0" 04:04:25 DEBUG }, 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "0" 04:04:25 DEBUG "0" 04:04:25 DEBUG ] 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG { 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "data": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "result": [ 04:04:25 DEBUG { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "instance": "10.142.0.107:9100" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "instance": "10.142.0.43:9100" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "0.00020000000000000757" 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG { 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "data": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "result": [ 04:04:25 DEBUG { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "0" 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG { 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "data": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "result": [ 04:04:25 DEBUG { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "0" 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713063865" 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG { 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "data": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "result": [ 04:04:25 DEBUG { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "2680.5" 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG { 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "data": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "result": [ 04:04:25 DEBUG { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "884547584" 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063865" 04:04:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063865" 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 INFO [loop_until]: kubectl --namespace=xlou top pods 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 DEBUG --- status code --- 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:04:25 INFO [http_cmd]: http status code OK 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG --- status code --- 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG { 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:04:25 DEBUG { 04:04:25 DEBUG { 04:04:25 DEBUG { 04:04:25 DEBUG { 04:04:25 DEBUG { 04:04:25 DEBUG { 04:04:25 DEBUG { 04:04:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:04:25 DEBUG { 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:04:25 DEBUG { 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG --- http response --- 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "data": { 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG { 04:04:25 DEBUG "data": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "data": { 04:04:25 DEBUG "data": { 04:04:25 DEBUG "data": { 04:04:25 DEBUG "data": { 04:04:25 DEBUG "data": { 04:04:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:04:25 DEBUG "data": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG { 04:04:25 DEBUG "data": { 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "data": { 04:04:25 DEBUG "result": [ 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG ----- output ----- 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "result": [ 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "data": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG { 04:04:25 DEBUG "result": [ 04:04:25 DEBUG "result": [ 04:04:25 DEBUG "result": [ 04:04:25 DEBUG "result": [ 04:04:25 DEBUG "result": [ 04:04:25 DEBUG { 04:04:25 DEBUG "result": [ 04:04:25 DEBUG { 04:04:25 DEBUG "data": { 04:04:25 DEBUG "result": [ 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "result": [ 04:04:25 DEBUG "metric": { 04:04:25 DEBUG { 04:04:25 DEBUG { 04:04:25 DEBUG { 04:04:25 DEBUG { 04:04:25 DEBUG { 04:04:25 DEBUG "status": "success", 04:04:25 DEBUG { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG { 04:04:25 DEBUG "result": [ 04:04:25 DEBUG { 04:04:25 DEBUG "pod": "ds-idrepo-1" 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "data": { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "instance": "10.142.0.107:9100" 04:04:25 DEBUG "result": [ 04:04:25 DEBUG "metric": { 04:04:25 DEBUG { 04:04:25 DEBUG "metric": { 04:04:25 DEBUG }, 04:04:25 DEBUG "pod": "ds-idrepo-1" 04:04:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:04:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:04:25 DEBUG "instance": "10.142.0.107:9100" 04:04:25 DEBUG "pod": "ds-idrepo-1" 04:04:25 DEBUG "resultType": "vector", 04:04:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:04:25 DEBUG }, 04:04:25 DEBUG { 04:04:25 DEBUG "pod": "ds-idrepo-1" 04:04:25 DEBUG "metric": { 04:04:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:04:25 DEBUG "value": [ 04:04:25 DEBUG }, 04:04:25 DEBUG }, 04:04:25 DEBUG }, 04:04:25 DEBUG }, 04:04:25 DEBUG }, 04:04:25 DEBUG "result": [ 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG "metric": { 04:04:25 DEBUG }, 04:04:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:04:25 DEBUG }, 04:04:25 DEBUG 04:04:25 DEBUG "value": [ 04:04:25 DEBUG "value": [ 04:04:25 DEBUG "value": [ 04:04:25 DEBUG "value": [ 04:04:25 DEBUG "value": [ 04:04:25 DEBUG { 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:04:25 DEBUG "value": [ 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "metric": { 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG }, 04:04:25 DEBUG 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "instance": "10.142.0.107:9100" 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG }, 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG [...] 04:04:25 DEBUG [...] 04:04:25 DEBUG [...] 04:04:25 DEBUG [...] 04:04:25 DEBUG "value": [ 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "pod": "ds-idrepo-0" 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG "instance": "10.142.0.43:9100" 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG 04:04:25 DEBUG }, 04:04:25 DEBUG "pod": "ds-idrepo-0" 04:04:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:04:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:04:25 DEBUG "instance": "10.142.0.43:9100" 04:04:25 DEBUG "pod": "ds-cts-1" 04:04:25 DEBUG [...] 04:04:25 DEBUG 04:04:25 DEBUG "pod": "ds-idrepo-1" 04:04:25 DEBUG 04:04:25 DEBUG "pod": "ds-cts-1" 04:04:25 DEBUG 04:04:25 DEBUG "pod": "idm-b85f9d978-j5pbm" 04:04:25 DEBUG "value": [ 04:04:25 DEBUG }, 04:04:25 DEBUG }, 04:04:25 DEBUG }, 04:04:25 DEBUG }, 04:04:25 DEBUG }, 04:04:25 DEBUG 04:04:25 DEBUG }, 04:04:25 DEBUG }, 04:04:25 DEBUG 04:04:25 DEBUG }, 04:04:25 DEBUG "pod": "ds-idrepo-1" 04:04:25 DEBUG }, 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 04:04:25 DEBUG "value": [ 04:04:25 DEBUG "value": [ 04:04:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:04:25 DEBUG "value": [ 04:04:25 DEBUG }, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG "0" 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG "7.894668237908247" 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "instance": "10.142.0.43:9100" 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG }, 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "value": [ 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG ] 04:04:25 DEBUG "0" 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG ] 04:04:25 DEBUG "12687858688" 04:04:25 DEBUG "0" 04:04:25 DEBUG }, 04:04:25 DEBUG "5471.233333333334" 04:04:25 DEBUG "0" 04:04:25 DEBUG "value": [ 04:04:25 DEBUG "0" 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG "1206.6420678487661" 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG "2681.5" 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG ] 04:04:25 DEBUG "value": [ 04:04:25 DEBUG ] 04:04:25 DEBUG ] 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG ] 04:04:25 DEBUG "0" 04:04:25 DEBUG ] 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG 1713063865, 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG "183498" 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG ] 04:04:25 DEBUG "0" 04:04:25 DEBUG ] 04:04:25 DEBUG ] 04:04:25 DEBUG ] 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG ] 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 DEBUG } 04:04:25 INFO [loop_until]: OK (rc = 0) 04:04:25 DEBUG --- stdout --- 04:04:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 11870m 8280Mi am-7d597cf7d6-8vr8f 11066m 7934Mi am-7d597cf7d6-9wz9h 14585m 8523Mi ds-cts-0 5481m 6945Mi ds-cts-1 5829m 7714Mi ds-cts-2 5688m 7730Mi ds-idrepo-0 13653m 4584Mi ds-idrepo-1 2132m 4302Mi ds-idrepo-2 1942m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 997Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 10m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7793m 749Mi 04:04:25 DEBUG --- stderr --- 04:04:25 DEBUG 04:04:25 INFO 04:04:25 INFO [loop_until]: kubectl --namespace=xlou top node 04:04:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:04:25 INFO [loop_until]: OK (rc = 0) 04:04:25 DEBUG --- stdout --- 04:04:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 4843Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5795m 19% 9944Mi 8% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12185m 40% 10316Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 21555m 72% 3034Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2234m 7% 6371Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 5543m 18% 10418Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2918Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 11437m 38% 9371Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 2082m 6% 5513Mi 4% gke-xlou-cdm-ds-0f601751-j488 5954m 19% 9272Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2223Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 15864m 53% 10047Mi 8% gke-xlou-cdm-ds-0f601751-nmxm 13864m 46% 6055Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 7791m 49% 4544Mi 7% 04:04:25 DEBUG --- stderr --- 04:04:25 DEBUG 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063925" 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "result": [ 04:05:25 DEBUG { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "instance": "10.142.0.107:9100" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "instance": "10.142.0.43:9100" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG "3867.3666666666663" 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "result": [ 04:05:25 DEBUG { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG "2680.266666666667" 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG { 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "result": [ 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "result": [ 04:05:25 DEBUG { 04:05:25 DEBUG "result": [ 04:05:25 DEBUG { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "pod": "ds-idrepo-1" 04:05:25 DEBUG }, 04:05:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:05:25 DEBUG 04:05:25 DEBUG "pod": "ds-idrepo-0" 04:05:25 DEBUG }, 04:05:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG "0.015" 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG "0" 04:05:25 DEBUG ] 04:05:25 DEBUG "0" 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 ERROR [http_cmd]: ERROR 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 ERROR --- status code --- 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- status code --- 04:05:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 ERROR --- http response --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:05:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:05:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "result": [ 04:05:25 DEBUG { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "pod": "ds-idrepo-1" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "pod": "ds-idrepo-0" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG "0" 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:05:25 DEBUG "result": [ 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG { 04:05:25 DEBUG { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:05:25 DEBUG "data": { 04:05:25 DEBUG }, 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "value": [ 04:05:25 DEBUG "result": [ 04:05:25 DEBUG 04:05:25 DEBUG { 04:05:25 DEBUG 04:05:25 DEBUG "metric": { 04:05:25 DEBUG 04:05:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:05:25 DEBUG [...] 04:05:25 DEBUG }, 04:05:25 DEBUG 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:05:25 DEBUG 04:05:25 DEBUG }, 04:05:25 DEBUG [...] 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG 04:05:25 DEBUG "0" 04:05:25 DEBUG 04:05:25 DEBUG ] 04:05:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:05:25 DEBUG } 04:05:25 DEBUG }, 04:05:25 DEBUG ] 04:05:25 DEBUG "value": [ 04:05:25 DEBUG } 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG } 04:05:25 DEBUG "2679.633333333333" 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063925" 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "result": [ 04:05:25 DEBUG { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "pod": "ds-idrepo-1" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "pod": "ds-idrepo-0" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG "0" 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "result": [ 04:05:25 DEBUG { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "instance": "10.142.0.107:9100" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "instance": "10.142.0.43:9100" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG "0.00026666666666651885" 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "result": [ 04:05:25 DEBUG { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG "0" 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "result": [ 04:05:25 DEBUG { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG "890564608" 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "result": [ 04:05:25 DEBUG { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG "343913.5" 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063925" 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "result": [ 04:05:25 DEBUG { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "instance": "10.142.0.107:9100" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "instance": "10.142.0.43:9100" 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG "5418.5" 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063925" 04:05:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063925" 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- status code --- 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- status code --- 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG --- http response --- 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:05:25 DEBUG --- http response --- 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG { 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG { 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG { 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG { 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG { 04:05:25 DEBUG "data": { 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:05:25 INFO [http_cmd]: http status code OK 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG "data": { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG "data": { 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG --- status code --- 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:05:25 DEBUG "data": { 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "data": { 04:05:25 DEBUG "result": [ 04:05:25 DEBUG { 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG { 04:05:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:05:25 DEBUG "result": [ 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "result": [ 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG { 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG "result": [ 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG --- http response --- 04:05:25 DEBUG { 04:05:25 DEBUG { 04:05:25 DEBUG "result": [ 04:05:25 DEBUG { 04:05:25 DEBUG "result": [ 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "data": { 04:05:25 DEBUG { 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:05:25 DEBUG "data": { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG { 04:05:25 DEBUG "pod": "ds-idrepo-1" 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "metric": { 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "pod": "ds-idrepo-1" 04:05:25 DEBUG ----- output ----- 04:05:25 DEBUG "data": { 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:05:25 DEBUG "metric": { 04:05:25 DEBUG }, 04:05:25 DEBUG "result": [ 04:05:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:05:25 DEBUG { 04:05:25 DEBUG "result": [ 04:05:25 DEBUG }, 04:05:25 DEBUG { 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "pod": "ds-idrepo-1" 04:05:25 DEBUG }, 04:05:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:05:25 DEBUG "value": [ 04:05:25 DEBUG { 04:05:25 DEBUG }, 04:05:25 DEBUG "status": "success", 04:05:25 INFO 04:05:25 DEBUG { 04:05:25 DEBUG "value": [ 04:05:25 DEBUG "status": "success", 04:05:25 DEBUG "result": [ 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG }, 04:05:25 DEBUG 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "value": [ 04:05:25 DEBUG "data": { 04:05:25 INFO [loop_until]: kubectl --namespace=xlou top pods 04:05:25 DEBUG "metric": { 04:05:25 DEBUG 04:05:25 DEBUG "data": { 04:05:25 DEBUG { 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG "pod": "ds-idrepo-1" 04:05:25 DEBUG 04:05:25 DEBUG "resultType": "vector", 04:05:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:05:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:05:25 DEBUG 04:05:25 DEBUG "resultType": "vector", 04:05:25 DEBUG "metric": { 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG }, 04:05:25 DEBUG 04:05:25 DEBUG "result": [ 04:05:25 DEBUG }, 04:05:25 DEBUG 04:05:25 DEBUG "result": [ 04:05:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG { 04:05:25 DEBUG "value": [ 04:05:25 DEBUG [...] 04:05:25 DEBUG { 04:05:25 DEBUG }, 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG "metric": { 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "metric": { 04:05:25 DEBUG "value": [ 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "instance": "10.142.0.107:9100" 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "instance": "10.142.0.107:9100" 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG }, 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG }, 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "pod": "ds-cts-1" 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG "value": [ 04:05:25 DEBUG [...] 04:05:25 DEBUG "pod": "ds-idrepo-0" 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:05:25 DEBUG 04:05:25 DEBUG }, 04:05:25 DEBUG 04:05:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG }, 04:05:25 DEBUG 04:05:25 DEBUG [...] 04:05:25 DEBUG "pod": "ds-idrepo-0" 04:05:25 DEBUG }, 04:05:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG }, 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG }, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG }, 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG 04:05:25 DEBUG 04:05:25 DEBUG "value": [ 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG "value": [ 04:05:25 DEBUG "0.000033333333333333335" 04:05:25 DEBUG "pod": "ds-cts-1" 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG [...] 04:05:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:05:25 DEBUG "0" 04:05:25 DEBUG [...] 04:05:25 DEBUG 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG "7.899653759563722" 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG ] 04:05:25 DEBUG }, 04:05:25 DEBUG "7.860282130544305" 04:05:25 DEBUG 04:05:25 DEBUG }, 04:05:25 DEBUG ] 04:05:25 DEBUG 04:05:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:05:25 DEBUG "0" 04:05:25 DEBUG ] 04:05:25 DEBUG "113071.96029776674" 04:05:25 DEBUG } 04:05:25 DEBUG "value": [ 04:05:25 DEBUG ] 04:05:25 DEBUG 04:05:25 DEBUG "value": [ 04:05:25 DEBUG } 04:05:25 DEBUG 04:05:25 DEBUG }, 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG ] 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG } 04:05:25 DEBUG 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG ] 04:05:25 DEBUG 04:05:25 DEBUG "value": [ 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG "0" 04:05:25 DEBUG ] 04:05:25 DEBUG "instance": "10.142.0.43:9100" 04:05:25 DEBUG "0" 04:05:25 DEBUG } 04:05:25 DEBUG "instance": "10.142.0.43:9100" 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG }, 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG }, 04:05:25 DEBUG "2679.866666666667" 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG "value": [ 04:05:25 DEBUG } 04:05:25 DEBUG "value": [ 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG ] 04:05:25 DEBUG 1713063925, 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG "0" 04:05:25 DEBUG } 04:05:25 DEBUG "12688350208" 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG ] 04:05:25 DEBUG ] 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 DEBUG } 04:05:25 INFO [loop_until]: OK (rc = 0) 04:05:25 DEBUG --- stdout --- 04:05:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 11866m 8282Mi am-7d597cf7d6-8vr8f 10770m 7934Mi am-7d597cf7d6-9wz9h 15020m 9102Mi ds-cts-0 5670m 6973Mi ds-cts-1 6013m 7703Mi ds-cts-2 5982m 7761Mi ds-idrepo-0 13835m 4584Mi ds-idrepo-1 2100m 4300Mi ds-idrepo-2 1991m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 996Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 14m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7838m 754Mi 04:05:25 DEBUG --- stderr --- 04:05:25 DEBUG 04:05:25 INFO 04:05:25 INFO [loop_until]: kubectl --namespace=xlou top node 04:05:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:05:25 INFO [loop_until]: OK (rc = 0) 04:05:25 DEBUG --- stdout --- 04:05:25 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4827Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5936m 19% 9991Mi 8% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12247m 41% 10318Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 21652m 72% 3016Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2237m 7% 6378Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 5763m 19% 10442Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2917Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 11515m 38% 9367Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 2116m 7% 5527Mi 4% gke-xlou-cdm-ds-0f601751-j488 6063m 20% 9261Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2213Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 15736m 52% 10618Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13890m 46% 6053Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8062m 50% 4544Mi 7% 04:05:25 DEBUG --- stderr --- 04:05:25 DEBUG 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063985" 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "0.015" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "0" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "instance": "10.142.0.107:9100" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "instance": "10.142.0.43:9100" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "2897.2666666666664" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "2669.766666666667" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "2637.7666666666664" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713063985" 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "ds-idrepo-1" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "ds-idrepo-0" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "0" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 ERROR [http_cmd]: ERROR 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:06:25 ERROR --- status code --- 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG ----- output ----- 04:06:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG { 04:06:25 ERROR --- http response --- 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:06:25 DEBUG "data": { 04:06:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:06:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "ds-idrepo-1" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "ds-idrepo-0" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "0" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063985" 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "instance": "10.142.0.107:9100" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "instance": "10.142.0.43:9100" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "0.00026666666666651885" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "ds-idrepo-1" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "ds-idrepo-0" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "0" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "2218.0333333333333" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "0" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713063985" 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "1117933568" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "467412" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063985" 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "instance": "10.142.0.107:9100" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "instance": "10.142.0.43:9100" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "5473.133333333334" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063985" 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "ds-idrepo-1" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "ds-cts-1" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "0.00030695833333333335" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "instance": "10.142.0.107:9100" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "instance": "10.142.0.43:9100" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "12688917504" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "ds-idrepo-1" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "ds-cts-1" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "0" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "7.903265242208621" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713063985" 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713063985" 04:06:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713063985" 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "ds-idrepo-1" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "ds-idrepo-0" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "0" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "3388532.438313724" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "ds-idrepo-1" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "ds-idrepo-0" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "0" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "2637.366666666667" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG { 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "0" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "7.92960241726826" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO [http_cmd]: http status code OK 04:06:25 DEBUG --- status code --- 04:06:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:06:25 DEBUG --- http response --- 04:06:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:06:25 DEBUG ----- output ----- 04:06:25 DEBUG { 04:06:25 DEBUG "status": "success", 04:06:25 DEBUG "data": { 04:06:25 DEBUG "resultType": "vector", 04:06:25 DEBUG "result": [ 04:06:25 DEBUG { 04:06:25 DEBUG "metric": { 04:06:25 DEBUG "instance": "10.142.0.107:9100" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG [...] 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG 04:06:25 DEBUG "instance": "10.142.0.43:9100" 04:06:25 DEBUG }, 04:06:25 DEBUG "value": [ 04:06:25 DEBUG 1713063985, 04:06:25 DEBUG "0" 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG ] 04:06:25 DEBUG } 04:06:25 DEBUG } 04:06:25 INFO 04:06:25 INFO [loop_until]: kubectl --namespace=xlou top pods 04:06:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:06:25 INFO [loop_until]: OK (rc = 0) 04:06:25 DEBUG --- stdout --- 04:06:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12333m 8294Mi am-7d597cf7d6-8vr8f 10963m 7937Mi am-7d597cf7d6-9wz9h 14657m 9124Mi ds-cts-0 6081m 6979Mi ds-cts-1 6192m 7708Mi ds-cts-2 7130m 7768Mi ds-idrepo-0 13893m 4584Mi ds-idrepo-1 2164m 4301Mi ds-idrepo-2 1980m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 996Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 11m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8064m 759Mi 04:06:25 DEBUG --- stderr --- 04:06:25 DEBUG 04:06:25 INFO 04:06:25 INFO [loop_until]: kubectl --namespace=xlou top node 04:06:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:06:26 INFO [loop_until]: OK (rc = 0) 04:06:26 DEBUG --- stdout --- 04:06:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4822Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 6421m 21% 9994Mi 8% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12697m 42% 10331Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 21872m 73% 3040Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2283m 7% 6380Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 6308m 21% 10450Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 58m 0% 2910Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12039m 40% 9378Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 2142m 7% 5512Mi 4% gke-xlou-cdm-ds-0f601751-j488 7421m 24% 9252Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 71m 0% 2213Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16239m 54% 10660Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14216m 47% 6041Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8110m 51% 4548Mi 7% 04:06:26 DEBUG --- stderr --- 04:06:26 DEBUG 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713064045" 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG { 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:07:25 DEBUG "data": { 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG }, 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "value": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:07:25 DEBUG 04:07:25 DEBUG "instance": "10.142.0.107:9100" 04:07:25 DEBUG }, 04:07:25 DEBUG 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG [...] 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG }, 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG 04:07:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:07:25 DEBUG "0.015" 04:07:25 DEBUG "instance": "10.142.0.43:9100" 04:07:25 DEBUG }, 04:07:25 DEBUG ] 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG } 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG ] 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "0" 04:07:25 DEBUG } 04:07:25 DEBUG "2722.9" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 ERROR [http_cmd]: ERROR 04:07:25 ERROR --- status code --- 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG --- status code --- 04:07:25 ERROR --- http response --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:07:25 DEBUG --- http response --- 04:07:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "ds-idrepo-1" 04:07:25 DEBUG "pod": "ds-idrepo-1" 04:07:25 DEBUG }, 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "pod": "ds-idrepo-0" 04:07:25 DEBUG "pod": "ds-idrepo-0" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "0" 04:07:25 DEBUG }, 04:07:25 DEBUG ] 04:07:25 DEBUG "value": [ 04:07:25 DEBUG } 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG ] 04:07:25 DEBUG "0" 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "1293.7666666666667" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "1289.4" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "2678.733333333333" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "instance": "10.142.0.107:9100" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "instance": "10.142.0.43:9100" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "0.00030000000000001136" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "ds-idrepo-1" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "pod": "ds-idrepo-0" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "0" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "0" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "instance": "10.142.0.107:9100" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "instance": "10.142.0.43:9100" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "12689503232" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "3018862592" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "426298.5" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 DEBUG "result": [ 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "instance": "10.142.0.107:9100" 04:07:25 DEBUG "data": { 04:07:25 DEBUG }, 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "value": [ 04:07:25 DEBUG "result": [ 04:07:25 DEBUG 04:07:25 DEBUG { 04:07:25 DEBUG 04:07:25 DEBUG "metric": { 04:07:25 DEBUG 04:07:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:07:25 DEBUG [...] 04:07:25 DEBUG }, 04:07:25 DEBUG 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "instance": "10.142.0.43:9100" 04:07:25 DEBUG 04:07:25 DEBUG }, 04:07:25 DEBUG [...] 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG 04:07:25 DEBUG "5327.366666666667" 04:07:25 DEBUG 04:07:25 DEBUG ] 04:07:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:07:25 DEBUG } 04:07:25 DEBUG }, 04:07:25 DEBUG ] 04:07:25 DEBUG "value": [ 04:07:25 DEBUG } 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG } 04:07:25 DEBUG "9.997364242432667" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "ds-idrepo-1" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "pod": "ds-cts-1" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "0.0001666666666666667" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "32659672.344293233" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "ds-idrepo-1" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "pod": "ds-idrepo-0" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "0" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "ds-idrepo-1" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "pod": "ds-cts-1" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "0" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064045" 04:07:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064045" 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "ds-idrepo-1" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "pod": "ds-idrepo-0" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "0" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "1293.3" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "0" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG { 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG "data": { 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG "result": [ 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 INFO [http_cmd]: http status code OK 04:07:25 DEBUG 04:07:25 DEBUG --- status code --- 04:07:25 DEBUG 04:07:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:07:25 DEBUG [...] 04:07:25 DEBUG --- http response --- 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG ----- output ----- 04:07:25 DEBUG "8.03689341319367" 04:07:25 DEBUG { 04:07:25 DEBUG ] 04:07:25 DEBUG "status": "success", 04:07:25 DEBUG } 04:07:25 DEBUG "data": { 04:07:25 DEBUG ] 04:07:25 DEBUG "resultType": "vector", 04:07:25 DEBUG } 04:07:25 DEBUG "result": [ 04:07:25 DEBUG } 04:07:25 DEBUG { 04:07:25 DEBUG "metric": { 04:07:25 DEBUG "instance": "10.142.0.107:9100" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG [...] 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG 04:07:25 DEBUG "instance": "10.142.0.43:9100" 04:07:25 DEBUG }, 04:07:25 DEBUG "value": [ 04:07:25 DEBUG 1713064045, 04:07:25 DEBUG "0" 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG ] 04:07:25 DEBUG } 04:07:25 DEBUG } 04:07:25 INFO 04:07:25 INFO [loop_until]: kubectl --namespace=xlou top pods 04:07:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:07:25 INFO [loop_until]: OK (rc = 0) 04:07:25 DEBUG --- stdout --- 04:07:25 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 10499m 8318Mi am-7d597cf7d6-8vr8f 8555m 7953Mi am-7d597cf7d6-9wz9h 13390m 9177Mi ds-cts-0 10840m 7603Mi ds-cts-1 11572m 7937Mi ds-cts-2 10239m 7811Mi ds-idrepo-0 10015m 4584Mi ds-idrepo-1 1463m 4301Mi ds-idrepo-2 1518m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 5m 996Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 8m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8046m 2787Mi 04:07:25 DEBUG --- stderr --- 04:07:25 DEBUG 04:07:26 INFO 04:07:26 INFO [loop_until]: kubectl --namespace=xlou top node 04:07:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:07:26 INFO [loop_until]: OK (rc = 0) 04:07:26 DEBUG --- stdout --- 04:07:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 74m 0% 4817Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10738m 35% 10007Mi 8% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9524m 31% 10363Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 12760m 42% 3618Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1440m 4% 6377Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 10548m 35% 11064Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2912Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 8982m 30% 9398Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1393m 4% 5513Mi 4% gke-xlou-cdm-ds-0f601751-j488 10239m 34% 9454Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 70m 0% 2214Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 14192m 47% 10751Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 9341m 31% 6062Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 7889m 49% 6373Mi 10% 04:07:26 DEBUG --- stderr --- 04:07:26 DEBUG 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064105" 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "ds-idrepo-1" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "ds-idrepo-0" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "0" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG { 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG "status": "success", 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713064105" 04:08:25 DEBUG "data": { 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "data": { 04:08:25 DEBUG "instance": "10.142.0.107:9100" 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG }, 04:08:25 DEBUG "result": [ 04:08:25 DEBUG "value": [ 04:08:25 DEBUG { 04:08:25 DEBUG 04:08:25 DEBUG "metric": { 04:08:25 DEBUG 04:08:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:08:25 DEBUG 04:08:25 DEBUG }, 04:08:25 DEBUG [...] 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "instance": "10.142.0.43:9100" 04:08:25 DEBUG [...] 04:08:25 DEBUG }, 04:08:25 DEBUG 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG 04:08:25 DEBUG "2808.1" 04:08:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:08:25 DEBUG ] 04:08:25 DEBUG }, 04:08:25 DEBUG } 04:08:25 DEBUG "value": [ 04:08:25 DEBUG ] 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG } 04:08:25 DEBUG "1333.5" 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG --- http response --- 04:08:25 ERROR [http_cmd]: ERROR 04:08:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:08:25 ERROR --- status code --- 04:08:25 DEBUG ----- output ----- 04:08:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:08:25 DEBUG { 04:08:25 ERROR --- http response --- 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "ds-idrepo-1" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "ds-idrepo-0" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "0" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "0.015" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "am-7d597cf7d6-8vr8f" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "0" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:08:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:08:25 DEBUG }, 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:08:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:08:25 DEBUG }, 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "2679.366666666667" 04:08:25 DEBUG "1333.8" 04:08:25 DEBUG ] 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064105" 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "ds-idrepo-1" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "ds-idrepo-0" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "0" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "instance": "10.142.0.107:9100" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "instance": "10.142.0.43:9100" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "0.00040000000000001514" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713064105" 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "5100572672" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "344703" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064105" 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "instance": "10.142.0.107:9100" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "instance": "10.142.0.43:9100" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "12690123776" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "0" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064105" 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "am-7d597cf7d6-8vr8f" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "5.760919786054471" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "instance": "10.142.0.107:9100" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "instance": "10.142.0.43:9100" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "5453.533333333334" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "ds-idrepo-1" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "ds-cts-1" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "0.00006666666666666667" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "am-7d597cf7d6-8vr8f" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "312842760.0891242" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713064105" 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "8.030657993067651" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "ds-idrepo-1" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "ds-idrepo-0" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "0" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "ds-idrepo-1" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "ds-cts-1" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "0" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064105" 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "instance": "10.142.0.107:9100" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "instance": "10.142.0.43:9100" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "0" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064105" 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:08:25 DEBUG 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG 04:08:25 DEBUG { 04:08:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG }, 04:08:25 DEBUG "data": { 04:08:25 DEBUG "value": [ 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "result": [ 04:08:25 DEBUG "1333.5333333333333" 04:08:25 DEBUG { 04:08:25 DEBUG ] 04:08:25 DEBUG "metric": { 04:08:25 DEBUG } 04:08:25 DEBUG "pod": "ds-idrepo-1" 04:08:25 DEBUG ] 04:08:25 DEBUG }, 04:08:25 DEBUG } 04:08:25 DEBUG "value": [ 04:08:25 DEBUG } 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "ds-idrepo-0" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "0" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO [http_cmd]: http status code OK 04:08:25 DEBUG --- status code --- 04:08:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:08:25 DEBUG --- http response --- 04:08:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:08:25 DEBUG ----- output ----- 04:08:25 DEBUG { 04:08:25 DEBUG "status": "success", 04:08:25 DEBUG "data": { 04:08:25 DEBUG "resultType": "vector", 04:08:25 DEBUG "result": [ 04:08:25 DEBUG { 04:08:25 DEBUG "metric": { 04:08:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG [...] 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG 04:08:25 DEBUG "pod": "am-7d597cf7d6-8vr8f" 04:08:25 DEBUG }, 04:08:25 DEBUG "value": [ 04:08:25 DEBUG 1713064105, 04:08:25 DEBUG "3979463.867019587" 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG ] 04:08:25 DEBUG } 04:08:25 DEBUG } 04:08:25 INFO 04:08:25 INFO [loop_until]: kubectl --namespace=xlou top pods 04:08:25 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:08:26 INFO [loop_until]: OK (rc = 0) 04:08:26 DEBUG --- stdout --- 04:08:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 8286m 9028Mi am-7d597cf7d6-8vr8f 7809m 8959Mi am-7d597cf7d6-9wz9h 10351m 9211Mi ds-cts-0 9501m 7578Mi ds-cts-1 10004m 7902Mi ds-cts-2 9453m 7813Mi ds-idrepo-0 7908m 4584Mi ds-idrepo-1 1211m 4301Mi ds-idrepo-2 1150m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 5m 996Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 7m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8085m 4773Mi 04:08:26 DEBUG --- stderr --- 04:08:26 DEBUG 04:08:26 INFO 04:08:26 INFO [loop_until]: kubectl --namespace=xlou top node 04:08:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:08:26 INFO [loop_until]: OK (rc = 0) 04:08:26 DEBUG --- stdout --- 04:08:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4820Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 9773m 32% 10006Mi 8% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8520m 28% 11074Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11193m 37% 3635Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1370m 4% 6375Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 9603m 32% 11033Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2914Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 8299m 27% 10395Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1273m 4% 5521Mi 4% gke-xlou-cdm-ds-0f601751-j488 9904m 33% 9448Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 2218Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 11247m 37% 10780Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 8538m 28% 6061Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8227m 51% 8737Mi 14% 04:08:26 DEBUG --- stderr --- 04:08:26 DEBUG 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "pod": "ds-idrepo-1" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "pod": "ds-idrepo-0" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "0" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG { 04:09:25 DEBUG { 04:09:25 DEBUG { 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "data": { 04:09:25 DEBUG "data": { 04:09:25 DEBUG "data": { 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG "result": [ 04:09:25 DEBUG "result": [ 04:09:25 DEBUG "result": [ 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG { 04:09:25 DEBUG { 04:09:25 DEBUG { 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:09:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:09:25 DEBUG "instance": "10.142.0.107:9100" 04:09:25 DEBUG "pod": "ds-idrepo-1" 04:09:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:09:25 DEBUG }, 04:09:25 DEBUG }, 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG }, 04:09:25 ERROR [http_cmd]: ERROR 04:09:25 DEBUG }, 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG "value": [ 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG "value": [ 04:09:25 ERROR --- status code --- 04:09:25 DEBUG "value": [ 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG 04:09:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG 04:09:25 ERROR --- http response --- 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:09:25 DEBUG 04:09:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:09:25 DEBUG 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG [...] 04:09:25 DEBUG [...] 04:09:25 DEBUG [...] 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG [...] 04:09:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:09:25 DEBUG [...] 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG { 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "data": { 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:09:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:09:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:09:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:09:25 DEBUG "instance": "10.142.0.43:9100" 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "pod": "ds-idrepo-0" 04:09:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG }, 04:09:25 DEBUG }, 04:09:25 DEBUG }, 04:09:25 DEBUG "result": [ 04:09:25 DEBUG }, 04:09:25 DEBUG }, 04:09:25 DEBUG { 04:09:25 DEBUG "value": [ 04:09:25 DEBUG "value": [ 04:09:25 DEBUG "value": [ 04:09:25 DEBUG { 04:09:25 DEBUG "value": [ 04:09:25 DEBUG "value": [ 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "metric": { 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "data": { 04:09:25 DEBUG "1285.5" 04:09:25 DEBUG "2495.366666666667" 04:09:25 DEBUG "2712.9" 04:09:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:09:25 DEBUG "0" 04:09:25 DEBUG "1293.1666666666667" 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG ] 04:09:25 DEBUG ] 04:09:25 DEBUG ] 04:09:25 DEBUG }, 04:09:25 DEBUG ] 04:09:25 DEBUG ] 04:09:25 DEBUG "result": [ 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 DEBUG "value": [ 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 DEBUG { 04:09:25 DEBUG ] 04:09:25 DEBUG ] 04:09:25 DEBUG ] 04:09:25 DEBUG 04:09:25 DEBUG ] 04:09:25 DEBUG ] 04:09:25 DEBUG "metric": { 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 DEBUG 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:09:25 DEBUG }, 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 DEBUG 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 DEBUG "value": [ 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:09:25 DEBUG 04:09:25 DEBUG }, 04:09:25 DEBUG 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:09:25 DEBUG "0.015" 04:09:25 DEBUG }, 04:09:25 DEBUG ] 04:09:25 DEBUG "value": [ 04:09:25 DEBUG } 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG ] 04:09:25 DEBUG "0" 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713064165" 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "pod": "ds-idrepo-1" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "pod": "ds-idrepo-0" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "0" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "8028934144" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "265115.5" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064165" 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "instance": "10.142.0.107:9100" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "instance": "10.142.0.43:9100" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "0.00026666666666651885" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "instance": "10.142.0.107:9100" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "instance": "10.142.0.43:9100" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "12690772992" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "0" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064165" 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "pod": "ds-idrepo-1" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "pod": "ds-idrepo-0" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "0" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "7.993032309066843" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "pod": "ds-idrepo-1" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "pod": "ds-cts-1" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "0.00016666666666666663" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "instance": "10.142.0.107:9100" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "instance": "10.142.0.43:9100" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "5340.5666666666675" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "18927220.30992417" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713064165" 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "instance": "10.142.0.107:9100" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "instance": "10.142.0.43:9100" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "0" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "8.038646579885452" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "pod": "ds-idrepo-1" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "pod": "ds-cts-1" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "0" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064165" 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:09:25 DEBUG "pod": "ds-idrepo-1" 04:09:25 DEBUG }, 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:09:25 DEBUG "pod": "ds-idrepo-0" 04:09:25 DEBUG }, 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "1285.9" 04:09:25 DEBUG "0" 04:09:25 DEBUG ] 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:25 INFO [http_cmd]: http status code OK 04:09:25 DEBUG --- status code --- 04:09:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:09:25 DEBUG --- http response --- 04:09:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:09:25 DEBUG ----- output ----- 04:09:25 DEBUG { 04:09:25 DEBUG "status": "success", 04:09:25 DEBUG "data": { 04:09:25 DEBUG "resultType": "vector", 04:09:25 DEBUG "result": [ 04:09:25 DEBUG { 04:09:25 DEBUG "metric": { 04:09:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG [...] 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG 04:09:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:09:25 DEBUG }, 04:09:25 DEBUG "value": [ 04:09:25 DEBUG 1713064165, 04:09:25 DEBUG "0" 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG ] 04:09:25 DEBUG } 04:09:25 DEBUG } 04:09:26 INFO 04:09:26 INFO [loop_until]: kubectl --namespace=xlou top pods 04:09:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:09:26 INFO [loop_until]: OK (rc = 0) 04:09:26 DEBUG --- stdout --- 04:09:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 8045m 8974Mi am-7d597cf7d6-8vr8f 7387m 8080Mi am-7d597cf7d6-9wz9h 10120m 9172Mi ds-cts-0 9492m 7597Mi ds-cts-1 9803m 7916Mi ds-cts-2 10034m 7826Mi ds-idrepo-0 7810m 4584Mi ds-idrepo-1 1225m 4300Mi ds-idrepo-2 1120m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 996Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 10m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7984m 7106Mi 04:09:26 DEBUG --- stderr --- 04:09:26 DEBUG 04:09:26 INFO 04:09:26 INFO [loop_until]: kubectl --namespace=xlou top node 04:09:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:09:26 INFO [loop_until]: OK (rc = 0) 04:09:26 DEBUG --- stdout --- 04:09:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4814Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10032m 33% 10017Mi 8% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8559m 28% 11028Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10788m 36% 3643Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1263m 4% 6383Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 9669m 32% 11023Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2915Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 8194m 27% 9524Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1224m 4% 5516Mi 4% gke-xlou-cdm-ds-0f601751-j488 9827m 32% 9449Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 70m 0% 2210Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 10672m 35% 10733Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 8160m 27% 6066Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 7916m 49% 10381Mi 17% 04:09:26 DEBUG --- stderr --- 04:09:26 DEBUG 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713064225" 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "1288.7" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 ERROR [http_cmd]: ERROR 04:10:25 DEBUG --- status code --- 04:10:25 ERROR --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 ERROR --- http response --- 04:10:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:10:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:10:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064225" 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "ds-idrepo-1" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "ds-idrepo-0" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "0" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "idm-b85f9d978-j5pbm" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "0" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "0.015" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG { 04:10:25 DEBUG { 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "data": { 04:10:25 DEBUG "data": { 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG "result": [ 04:10:25 DEBUG "result": [ 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG { 04:10:25 DEBUG { 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "ds-idrepo-1" 04:10:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:10:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:10:25 DEBUG "instance": "10.142.0.107:9100" 04:10:25 DEBUG }, 04:10:25 DEBUG }, 04:10:25 DEBUG }, 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG "value": [ 04:10:25 DEBUG "value": [ 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG [...] 04:10:25 DEBUG [...] 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "ds-idrepo-0" 04:10:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:10:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:10:25 DEBUG "instance": "10.142.0.43:9100" 04:10:25 DEBUG }, 04:10:25 DEBUG }, 04:10:25 DEBUG }, 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG "value": [ 04:10:25 DEBUG "value": [ 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "0" 04:10:25 DEBUG "883.7333333333333" 04:10:25 DEBUG "883.7333333333333" 04:10:25 DEBUG "4368.666666666667" 04:10:25 DEBUG ] 04:10:25 DEBUG ] 04:10:25 DEBUG ] 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG ] 04:10:25 DEBUG ] 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "ds-idrepo-1" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "ds-idrepo-0" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "0" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713064225" 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "8501276672" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "224870" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064225" 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "instance": "10.142.0.107:9100" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "instance": "10.142.0.43:9100" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "0.0002666666666667557" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064225" 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "instance": "10.142.0.107:9100" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "instance": "10.142.0.43:9100" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "12691266560" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "0" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "ds-idrepo-1" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "ds-idrepo-0" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "0" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064225" 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "0" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG { 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG "data": { 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG "result": [ 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG { 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "pod": "ds-idrepo-1" 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG }, 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG "value": [ 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:10:25 DEBUG 04:10:25 DEBUG "instance": "10.142.0.107:9100" 04:10:25 DEBUG }, 04:10:25 DEBUG [...] 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "ds-cts-1" 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG }, 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "0.00006666666666666667" 04:10:25 DEBUG 04:10:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:10:25 DEBUG ] 04:10:25 DEBUG "instance": "10.142.0.43:9100" 04:10:25 DEBUG }, 04:10:25 DEBUG } 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG ] 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG } 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "0" 04:10:25 DEBUG } 04:10:25 DEBUG "5213.266666666667" 04:10:25 DEBUG ] 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064225" 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713064225" 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "instance": "10.142.0.107:9100" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "instance": "10.142.0.43:9100" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "0" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "8.030452847470382" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "ds-idrepo-1" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "ds-cts-1" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "0" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064225" 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "883.7333333333333" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 INFO [http_cmd]: http status code OK 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG --- status code --- 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG --- http response --- 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "ds-idrepo-1" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "ds-idrepo-0" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "0" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:10:25 DEBUG ----- output ----- 04:10:25 DEBUG { 04:10:25 DEBUG "status": "success", 04:10:25 DEBUG "data": { 04:10:25 DEBUG "resultType": "vector", 04:10:25 DEBUG "result": [ 04:10:25 DEBUG { 04:10:25 DEBUG "metric": { 04:10:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG [...] 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG 04:10:25 DEBUG "pod": "idm-b85f9d978-j5pbm" 04:10:25 DEBUG }, 04:10:25 DEBUG "value": [ 04:10:25 DEBUG 1713064225, 04:10:25 DEBUG "925.6002241482188" 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG ] 04:10:25 DEBUG } 04:10:25 DEBUG } 04:10:26 INFO 04:10:26 INFO [loop_until]: kubectl --namespace=xlou top pods 04:10:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:10:26 INFO [loop_until]: OK (rc = 0) 04:10:26 DEBUG --- stdout --- 04:10:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6752m 8975Mi am-7d597cf7d6-8vr8f 6449m 8082Mi am-7d597cf7d6-9wz9h 8714m 8822Mi ds-cts-0 6779m 7589Mi ds-cts-1 7244m 7898Mi ds-cts-2 7378m 7823Mi ds-idrepo-0 7256m 4583Mi ds-idrepo-1 1083m 4300Mi ds-idrepo-2 1043m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 997Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 9m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8122m 8013Mi 04:10:26 DEBUG --- stderr --- 04:10:26 DEBUG 04:10:26 INFO 04:10:26 INFO [loop_until]: kubectl --namespace=xlou top node 04:10:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:10:26 INFO [loop_until]: OK (rc = 0) 04:10:26 DEBUG --- stdout --- 04:10:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 59m 0% 4821Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 6747m 22% 10014Mi 8% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6914m 23% 11030Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10456m 35% 3623Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1183m 3% 6369Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 6580m 22% 11024Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2912Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 6695m 22% 9535Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1138m 3% 5530Mi 4% gke-xlou-cdm-ds-0f601751-j488 6148m 20% 9454Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 71m 0% 2219Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 9434m 31% 10388Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 7462m 24% 6067Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8199m 51% 11005Mi 18% 04:10:26 DEBUG --- stderr --- 04:10:26 DEBUG 04:10:36 WARNING File /tmp/lodemon_data-20240414_035725/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [14/Apr/2024 04:10:36] "GET /monitoring/average?start_time=20240414_030420&stop_time=20240414_030835 HTTP/1.1" 200 - 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "1333.1333333333332" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713064285" 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 ERROR [http_cmd]: ERROR 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 ERROR --- status code --- 04:11:25 DEBUG --- status code --- 04:11:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 ERROR --- http response --- 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:11:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "pod": "ds-idrepo-1" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "pod": "ds-idrepo-0" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "0" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713064285" 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "807.8666666666667" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "pod": "ds-idrepo-1" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "pod": "ds-idrepo-0" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "0" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "512991232" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "171422.5" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064285" 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG --- status code --- 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG { 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "data": { 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "pod": "ds-idrepo-1" 04:11:25 DEBUG "result": [ 04:11:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:11:25 DEBUG }, 04:11:25 DEBUG { 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG }, 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG "pod": "ds-idrepo-0" 04:11:25 DEBUG 04:11:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:11:25 DEBUG }, 04:11:25 DEBUG 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "0" 04:11:25 DEBUG }, 04:11:25 DEBUG "0.015" 04:11:25 DEBUG ] 04:11:25 DEBUG "value": [ 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG "0" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "instance": "10.142.0.107:9100" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "instance": "10.142.0.43:9100" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "2913.7" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "806.4666666666667" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064285" 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "instance": "10.142.0.107:9100" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "instance": "10.142.0.43:9100" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "12691874816" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064285" 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "instance": "10.142.0.107:9100" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "instance": "10.142.0.43:9100" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "0.0002666666666667557" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "0" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "3.844320800613366" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "pod": "ds-idrepo-1" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "pod": "ds-idrepo-0" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "0" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "pod": "ds-idrepo-1" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "pod": "ds-cts-1" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "0" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064285" 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064285" 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "pod": "ds-idrepo-1" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "pod": "ds-cts-1" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "0" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "instance": "10.142.0.107:9100" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "instance": "10.142.0.43:9100" 04:11:25 DEBUG }, 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064285" 04:11:25 DEBUG "value": [ 04:11:25 INFO [http_cmd]: http status code OK 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "5206.6" 04:11:25 DEBUG --- status code --- 04:11:25 DEBUG ] 04:11:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:25 DEBUG } 04:11:25 DEBUG --- http response --- 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:11:25 DEBUG ----- output ----- 04:11:25 DEBUG { 04:11:25 DEBUG "status": "success", 04:11:25 DEBUG "data": { 04:11:25 DEBUG "resultType": "vector", 04:11:25 DEBUG "result": [ 04:11:25 DEBUG { 04:11:25 DEBUG "metric": { 04:11:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG [...] 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG 04:11:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:11:25 DEBUG }, 04:11:25 DEBUG "value": [ 04:11:25 DEBUG 1713064285, 04:11:25 DEBUG "110573.91396314217" 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG ] 04:11:25 DEBUG } 04:11:25 DEBUG } 04:11:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713064285" 04:11:26 INFO [http_cmd]: http status code OK 04:11:26 DEBUG --- status code --- 04:11:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:26 DEBUG --- http response --- 04:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:11:26 DEBUG ----- output ----- 04:11:26 DEBUG { 04:11:26 DEBUG "status": "success", 04:11:26 DEBUG "data": { 04:11:26 DEBUG "resultType": "vector", 04:11:26 DEBUG "result": [ 04:11:26 DEBUG { 04:11:26 DEBUG "metric": { 04:11:26 DEBUG "instance": "10.142.0.107:9100" 04:11:26 DEBUG }, 04:11:26 DEBUG "value": [ 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG [...] 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG "instance": "10.142.0.43:9100" 04:11:26 DEBUG }, 04:11:26 DEBUG "value": [ 04:11:26 DEBUG 1713064285, 04:11:26 DEBUG "0" 04:11:26 DEBUG ] 04:11:26 DEBUG } 04:11:26 DEBUG ] 04:11:26 DEBUG } 04:11:26 DEBUG } 04:11:26 INFO [http_cmd]: http status code OK 04:11:26 DEBUG --- status code --- 04:11:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:26 DEBUG --- http response --- 04:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:11:26 DEBUG ----- output ----- 04:11:26 DEBUG { 04:11:26 DEBUG "status": "success", 04:11:26 DEBUG "data": { 04:11:26 DEBUG "resultType": "vector", 04:11:26 DEBUG "result": [ 04:11:26 DEBUG { 04:11:26 DEBUG "metric": { 04:11:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:11:26 DEBUG }, 04:11:26 DEBUG "value": [ 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG [...] 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:11:26 DEBUG }, 04:11:26 DEBUG "value": [ 04:11:26 DEBUG 1713064285, 04:11:26 DEBUG "1.6884080809399569" 04:11:26 DEBUG ] 04:11:26 DEBUG } 04:11:26 DEBUG ] 04:11:26 DEBUG } 04:11:26 DEBUG } 04:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064286" 04:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064286" 04:11:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064286" 04:11:26 INFO [http_cmd]: http status code OK 04:11:26 DEBUG --- status code --- 04:11:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:26 DEBUG --- http response --- 04:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:11:26 DEBUG ----- output ----- 04:11:26 DEBUG { 04:11:26 DEBUG "status": "success", 04:11:26 DEBUG "data": { 04:11:26 DEBUG "resultType": "vector", 04:11:26 DEBUG "result": [ 04:11:26 DEBUG { 04:11:26 DEBUG "metric": { 04:11:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:11:26 DEBUG }, 04:11:26 DEBUG "value": [ 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG [...] 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:11:26 DEBUG }, 04:11:26 DEBUG "value": [ 04:11:26 DEBUG 1713064286, 04:11:26 DEBUG "807.7" 04:11:26 DEBUG ] 04:11:26 DEBUG } 04:11:26 DEBUG ] 04:11:26 DEBUG } 04:11:26 DEBUG } 04:11:26 INFO [http_cmd]: http status code OK 04:11:26 DEBUG --- status code --- 04:11:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:26 DEBUG --- http response --- 04:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:11:26 DEBUG ----- output ----- 04:11:26 DEBUG { 04:11:26 DEBUG "status": "success", 04:11:26 DEBUG "data": { 04:11:26 DEBUG "resultType": "vector", 04:11:26 DEBUG "result": [ 04:11:26 DEBUG { 04:11:26 DEBUG "metric": { 04:11:26 DEBUG "pod": "ds-idrepo-1" 04:11:26 DEBUG }, 04:11:26 DEBUG "value": [ 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG [...] 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG "pod": "ds-idrepo-0" 04:11:26 DEBUG }, 04:11:26 DEBUG "value": [ 04:11:26 DEBUG 1713064286, 04:11:26 DEBUG "0" 04:11:26 DEBUG ] 04:11:26 DEBUG } 04:11:26 DEBUG ] 04:11:26 DEBUG } 04:11:26 DEBUG } 04:11:26 INFO [http_cmd]: http status code OK 04:11:26 DEBUG --- status code --- 04:11:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:11:26 DEBUG --- http response --- 04:11:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:11:26 DEBUG ----- output ----- 04:11:26 DEBUG { 04:11:26 DEBUG "status": "success", 04:11:26 DEBUG "data": { 04:11:26 DEBUG "resultType": "vector", 04:11:26 DEBUG "result": [ 04:11:26 DEBUG { 04:11:26 DEBUG "metric": { 04:11:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:11:26 DEBUG }, 04:11:26 DEBUG "value": [ 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG [...] 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG 04:11:26 DEBUG "pod": "ds-cts-0" 04:11:26 DEBUG }, 04:11:26 DEBUG "value": [ 04:11:26 DEBUG 1713064286, 04:11:26 DEBUG "11421216.12152972" 04:11:26 DEBUG ] 04:11:26 DEBUG } 04:11:26 DEBUG ] 04:11:26 DEBUG } 04:11:26 DEBUG } 04:11:26 INFO 04:11:26 INFO [loop_until]: kubectl --namespace=xlou top pods 04:11:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:11:26 INFO [loop_until]: OK (rc = 0) 04:11:26 DEBUG --- stdout --- 04:11:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 1208m 8971Mi am-7d597cf7d6-8vr8f 1114m 8081Mi am-7d597cf7d6-9wz9h 1481m 8809Mi ds-cts-0 2758m 7549Mi ds-cts-1 2942m 7927Mi ds-cts-2 2964m 7817Mi ds-idrepo-0 731m 4583Mi ds-idrepo-1 107m 4301Mi ds-idrepo-2 102m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 997Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 13m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1087m 396Mi 04:11:26 DEBUG --- stderr --- 04:11:26 DEBUG 04:11:26 INFO 04:11:26 INFO [loop_until]: kubectl --namespace=xlou top node 04:11:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:11:26 INFO [loop_until]: OK (rc = 0) 04:11:26 DEBUG --- stdout --- 04:11:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4829Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2977m 9% 10020Mi 8% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1237m 4% 11055Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3058Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 152m 0% 6381Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 2939m 9% 11023Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 57m 0% 2928Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1274m 4% 9548Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 154m 0% 5528Mi 4% gke-xlou-cdm-ds-0f601751-j488 2973m 9% 9437Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 70m 0% 2208Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 1650m 5% 10417Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 796m 2% 6057Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1598m 10% 4250Mi 7% 04:11:26 DEBUG --- stderr --- 04:11:26 DEBUG 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064345" 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064345" 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064345" 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064345" 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064345" 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713064345" 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713064345" 04:12:25 INFO [http_cmd]: http status code OK 04:12:25 DEBUG --- status code --- 04:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:25 DEBUG --- http response --- 04:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:12:25 DEBUG ----- output ----- 04:12:25 DEBUG { 04:12:25 DEBUG "status": "success", 04:12:25 DEBUG "data": { 04:12:25 DEBUG "resultType": "vector", 04:12:25 DEBUG "result": [ 04:12:25 DEBUG { 04:12:25 DEBUG "metric": { 04:12:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG [...] 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 1713064345, 04:12:25 DEBUG "1287.1333333333332" 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 INFO [http_cmd]: http status code OK 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064345" 04:12:25 INFO [http_cmd]: http status code OK 04:12:25 DEBUG --- status code --- 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064345" 04:12:25 ERROR [http_cmd]: ERROR 04:12:25 INFO [http_cmd]: http status code OK 04:12:25 INFO [http_cmd]: http status code OK 04:12:25 DEBUG --- status code --- 04:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:25 ERROR --- status code --- 04:12:25 DEBUG --- status code --- 04:12:25 DEBUG --- status code --- 04:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:25 DEBUG --- http response --- 04:12:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:25 DEBUG --- http response --- 04:12:25 ERROR --- http response --- 04:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:12:25 DEBUG --- http response --- 04:12:25 DEBUG --- http response --- 04:12:25 INFO [http_cmd]: http status code OK 04:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:12:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:12:25 DEBUG ----- output ----- 04:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:12:25 DEBUG --- status code --- 04:12:25 DEBUG ----- output ----- 04:12:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:12:25 DEBUG { 04:12:25 DEBUG ----- output ----- 04:12:25 DEBUG ----- output ----- 04:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:25 DEBUG { 04:12:25 DEBUG "status": "success", 04:12:25 DEBUG { 04:12:25 DEBUG { 04:12:25 DEBUG --- http response --- 04:12:25 DEBUG "status": "success", 04:12:25 DEBUG "data": { 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064345" 04:12:25 DEBUG "status": "success", 04:12:25 DEBUG "status": "success", 04:12:25 DEBUG "data": { 04:12:25 DEBUG "resultType": "vector", 04:12:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:12:25 DEBUG "data": { 04:12:25 DEBUG "data": { 04:12:25 DEBUG "resultType": "vector", 04:12:25 DEBUG "result": [ 04:12:25 DEBUG "resultType": "vector", 04:12:25 DEBUG "resultType": "vector", 04:12:25 DEBUG "result": [ 04:12:25 DEBUG { 04:12:25 DEBUG "result": [ 04:12:25 DEBUG "result": [ 04:12:25 DEBUG { 04:12:25 DEBUG "metric": { 04:12:25 DEBUG { 04:12:25 DEBUG { 04:12:25 DEBUG "metric": { 04:12:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:12:25 DEBUG "metric": { 04:12:25 DEBUG "metric": { 04:12:25 DEBUG "pod": "ds-idrepo-1" 04:12:25 DEBUG }, 04:12:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:12:25 DEBUG "pod": "ds-idrepo-1" 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG }, 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 04:12:25 DEBUG "value": [ 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG [...] 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG [...] 04:12:25 DEBUG 04:12:25 DEBUG [...] 04:12:25 DEBUG [...] 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG "pod": "ds-idrepo-0" 04:12:25 DEBUG }, 04:12:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:12:25 DEBUG "pod": "ds-idrepo-0" 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG }, 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 1713064345, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 1713064345, 04:12:25 DEBUG "3071860736" 04:12:25 DEBUG 1713064345, 04:12:25 DEBUG 1713064345, 04:12:25 DEBUG "0" 04:12:25 DEBUG ] 04:12:25 DEBUG "199485" 04:12:25 DEBUG "0" 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG ] 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG ] 04:12:25 INFO [http_cmd]: http status code OK 04:12:25 INFO [http_cmd]: http status code OK 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG --- status code --- 04:12:25 DEBUG --- status code --- 04:12:25 DEBUG ] 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 DEBUG --- http response --- 04:12:25 DEBUG --- http response --- 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:12:25 DEBUG ----- output ----- 04:12:25 DEBUG ----- output ----- 04:12:25 DEBUG { 04:12:25 DEBUG { 04:12:25 DEBUG "status": "success", 04:12:25 DEBUG "status": "success", 04:12:25 DEBUG "data": { 04:12:25 DEBUG "data": { 04:12:25 DEBUG "resultType": "vector", 04:12:25 DEBUG "resultType": "vector", 04:12:25 DEBUG "result": [ 04:12:25 DEBUG "result": [ 04:12:25 DEBUG { 04:12:25 DEBUG { 04:12:25 DEBUG "metric": { 04:12:25 DEBUG "metric": { 04:12:25 DEBUG "instance": "10.142.0.107:9100" 04:12:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:12:25 DEBUG }, 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG [...] 04:12:25 DEBUG [...] 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG "instance": "10.142.0.43:9100" 04:12:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:12:25 DEBUG }, 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 1713064345, 04:12:25 DEBUG 1713064345, 04:12:25 DEBUG "2809.333333333333" 04:12:25 DEBUG "1381.4" 04:12:25 DEBUG ] 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 DEBUG ] 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 INFO [http_cmd]: http status code OK 04:12:25 DEBUG --- status code --- 04:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:25 DEBUG --- http response --- 04:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:12:25 DEBUG ----- output ----- 04:12:25 DEBUG { 04:12:25 DEBUG "status": "success", 04:12:25 DEBUG "data": { 04:12:25 DEBUG "resultType": "vector", 04:12:25 DEBUG "result": [ 04:12:25 DEBUG { 04:12:25 DEBUG "metric": { 04:12:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG [...] 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG "pod": "idm-b85f9d978-j5pbm" 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 1713064345, 04:12:25 DEBUG "0" 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064345" 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064345" 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713064345" 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064345" 04:12:25 INFO [http_cmd]: http status code OK 04:12:25 DEBUG --- status code --- 04:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:25 DEBUG --- http response --- 04:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:12:25 DEBUG ----- output ----- 04:12:25 DEBUG { 04:12:25 DEBUG "status": "success", 04:12:25 DEBUG "data": { 04:12:25 DEBUG "resultType": "vector", 04:12:25 DEBUG "result": [ 04:12:25 DEBUG { 04:12:25 DEBUG "metric": { 04:12:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG [...] 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 1713064345, 04:12:25 DEBUG "0.015" 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 INFO [http_cmd]: http status code OK 04:12:25 INFO [http_cmd]: http status code OK 04:12:25 INFO [http_cmd]: http status code OK 04:12:25 DEBUG --- status code --- 04:12:25 DEBUG --- status code --- 04:12:25 DEBUG --- status code --- 04:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:25 DEBUG --- http response --- 04:12:25 DEBUG --- http response --- 04:12:25 DEBUG --- http response --- 04:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:12:25 DEBUG ----- output ----- 04:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:12:25 DEBUG ----- output ----- 04:12:25 DEBUG { 04:12:25 DEBUG ----- output ----- 04:12:25 DEBUG { 04:12:25 DEBUG "status": "success", 04:12:25 DEBUG { 04:12:25 DEBUG "status": "success", 04:12:25 DEBUG "data": { 04:12:25 DEBUG "status": "success", 04:12:25 DEBUG "data": { 04:12:25 DEBUG "resultType": "vector", 04:12:25 DEBUG "data": { 04:12:25 DEBUG "resultType": "vector", 04:12:25 DEBUG "result": [ 04:12:25 DEBUG "resultType": "vector", 04:12:25 DEBUG "result": [ 04:12:25 DEBUG { 04:12:25 DEBUG "result": [ 04:12:25 DEBUG { 04:12:25 DEBUG "metric": { 04:12:25 DEBUG { 04:12:25 DEBUG "metric": { 04:12:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:12:25 DEBUG "metric": { 04:12:25 DEBUG "pod": "ds-idrepo-1" 04:12:25 DEBUG }, 04:12:25 DEBUG "instance": "10.142.0.107:9100" 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG [...] 04:12:25 DEBUG 04:12:25 DEBUG [...] 04:12:25 DEBUG 04:12:25 DEBUG [...] 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:12:25 DEBUG 04:12:25 DEBUG "pod": "ds-idrepo-0" 04:12:25 DEBUG }, 04:12:25 DEBUG "instance": "10.142.0.43:9100" 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 1713064345, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 1713064345, 04:12:25 DEBUG "1381.4666666666667" 04:12:25 DEBUG 1713064345, 04:12:25 DEBUG "0" 04:12:25 DEBUG ] 04:12:25 DEBUG "12692448256" 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064345" 04:12:25 INFO [http_cmd]: http status code OK 04:12:25 DEBUG --- status code --- 04:12:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:25 DEBUG --- http response --- 04:12:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:12:25 DEBUG ----- output ----- 04:12:25 DEBUG { 04:12:25 DEBUG "status": "success", 04:12:25 DEBUG "data": { 04:12:25 DEBUG "resultType": "vector", 04:12:25 DEBUG "result": [ 04:12:25 DEBUG { 04:12:25 DEBUG "metric": { 04:12:25 DEBUG "instance": "10.142.0.107:9100" 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG [...] 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG 04:12:25 DEBUG "instance": "10.142.0.43:9100" 04:12:25 DEBUG }, 04:12:25 DEBUG "value": [ 04:12:25 DEBUG 1713064345, 04:12:25 DEBUG "0.00023333333333350008" 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG ] 04:12:25 DEBUG } 04:12:25 DEBUG } 04:12:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064345" 04:12:26 INFO [http_cmd]: http status code OK 04:12:26 DEBUG --- status code --- 04:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:26 DEBUG --- http response --- 04:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:12:26 DEBUG ----- output ----- 04:12:26 DEBUG { 04:12:26 DEBUG "status": "success", 04:12:26 DEBUG "data": { 04:12:26 DEBUG "resultType": "vector", 04:12:26 DEBUG "result": [ 04:12:26 DEBUG { 04:12:26 DEBUG "metric": { 04:12:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG [...] 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 1713064345, 04:12:26 DEBUG "0" 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG } 04:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064346" 04:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064346" 04:12:26 INFO [http_cmd]: http status code OK 04:12:26 DEBUG --- status code --- 04:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:26 DEBUG --- http response --- 04:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:12:26 DEBUG ----- output ----- 04:12:26 DEBUG { 04:12:26 DEBUG "status": "success", 04:12:26 DEBUG "data": { 04:12:26 DEBUG "resultType": "vector", 04:12:26 DEBUG "result": [ 04:12:26 DEBUG { 04:12:26 DEBUG "metric": { 04:12:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG [...] 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 1713064346, 04:12:26 DEBUG "0" 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG } 04:12:26 INFO [http_cmd]: http status code OK 04:12:26 DEBUG --- status code --- 04:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:26 DEBUG --- http response --- 04:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:12:26 DEBUG ----- output ----- 04:12:26 DEBUG { 04:12:26 DEBUG "status": "success", 04:12:26 DEBUG "data": { 04:12:26 DEBUG "resultType": "vector", 04:12:26 DEBUG "result": [ 04:12:26 DEBUG { 04:12:26 DEBUG "metric": { 04:12:26 DEBUG "pod": "ds-idrepo-1" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG [...] 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG "pod": "ds-idrepo-0" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 1713064346, 04:12:26 DEBUG "0" 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG } 04:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713064346" 04:12:26 INFO [http_cmd]: http status code OK 04:12:26 DEBUG --- status code --- 04:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:26 DEBUG --- http response --- 04:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:12:26 DEBUG ----- output ----- 04:12:26 DEBUG { 04:12:26 DEBUG "status": "success", 04:12:26 DEBUG "data": { 04:12:26 DEBUG "resultType": "vector", 04:12:26 DEBUG "result": [ 04:12:26 DEBUG { 04:12:26 DEBUG "metric": { 04:12:26 DEBUG "pod": "ds-idrepo-1" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG [...] 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG "pod": "ds-cts-1" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 1713064346, 04:12:26 DEBUG "0" 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG } 04:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064346" 04:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064346" 04:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064346" 04:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064346" 04:12:26 INFO [http_cmd]: http status code OK 04:12:26 DEBUG --- status code --- 04:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:26 DEBUG --- http response --- 04:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:12:26 DEBUG ----- output ----- 04:12:26 DEBUG { 04:12:26 DEBUG "status": "success", 04:12:26 DEBUG "data": { 04:12:26 DEBUG "resultType": "vector", 04:12:26 DEBUG "result": [ 04:12:26 DEBUG { 04:12:26 DEBUG "metric": { 04:12:26 DEBUG "pod": "ds-idrepo-1" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG [...] 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG "pod": "ds-cts-1" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 1713064346, 04:12:26 DEBUG "0.000050861111111111116" 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG } 04:12:26 INFO [http_cmd]: http status code OK 04:12:26 DEBUG --- status code --- 04:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:26 DEBUG --- http response --- 04:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:12:26 DEBUG ----- output ----- 04:12:26 DEBUG { 04:12:26 DEBUG "status": "success", 04:12:26 DEBUG "data": { 04:12:26 DEBUG "resultType": "vector", 04:12:26 DEBUG "result": [ 04:12:26 DEBUG { 04:12:26 DEBUG "metric": { 04:12:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG [...] 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 1713064346, 04:12:26 DEBUG "1381.4" 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG } 04:12:26 INFO [http_cmd]: http status code OK 04:12:26 DEBUG --- status code --- 04:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:26 DEBUG --- http response --- 04:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064346" 04:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:12:26 DEBUG ----- output ----- 04:12:26 DEBUG { 04:12:26 DEBUG "status": "success", 04:12:26 DEBUG "data": { 04:12:26 DEBUG "resultType": "vector", 04:12:26 DEBUG "result": [ 04:12:26 DEBUG { 04:12:26 DEBUG "metric": { 04:12:26 DEBUG "instance": "10.142.0.107:9100" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 INFO [http_cmd]: http status code OK 04:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713064346" 04:12:26 DEBUG 04:12:26 DEBUG --- status code --- 04:12:26 DEBUG 04:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:26 DEBUG 04:12:26 DEBUG --- http response --- 04:12:26 DEBUG [...] 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG "instance": "10.142.0.43:9100" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 1713064346, 04:12:26 DEBUG "5493.466666666667" 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG } 04:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:12:26 DEBUG ----- output ----- 04:12:26 DEBUG { 04:12:26 DEBUG "status": "success", 04:12:26 DEBUG "data": { 04:12:26 DEBUG "resultType": "vector", 04:12:26 DEBUG "result": [ 04:12:26 DEBUG { 04:12:26 DEBUG "metric": { 04:12:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG [...] 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 1713064346, 04:12:26 DEBUG "0" 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG } 04:12:26 INFO [http_cmd]: http status code OK 04:12:26 DEBUG --- status code --- 04:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:26 DEBUG --- http response --- 04:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:12:26 DEBUG ----- output ----- 04:12:26 DEBUG { 04:12:26 DEBUG "status": "success", 04:12:26 DEBUG "data": { 04:12:26 DEBUG "resultType": "vector", 04:12:26 DEBUG "result": [ 04:12:26 DEBUG { 04:12:26 DEBUG "metric": { 04:12:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG [...] 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 1713064346, 04:12:26 DEBUG "8.020595457539871" 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG } 04:12:26 INFO [http_cmd]: http status code OK 04:12:26 DEBUG --- status code --- 04:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:26 DEBUG --- http response --- 04:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:12:26 DEBUG ----- output ----- 04:12:26 DEBUG { 04:12:26 DEBUG "status": "success", 04:12:26 DEBUG "data": { 04:12:26 DEBUG "resultType": "vector", 04:12:26 DEBUG "result": [ 04:12:26 DEBUG { 04:12:26 DEBUG "metric": { 04:12:26 DEBUG "instance": "10.142.0.107:9100" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG [...] 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG "instance": "10.142.0.43:9100" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 1713064346, 04:12:26 DEBUG "0" 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG } 04:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064346" 04:12:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064346" 04:12:26 INFO [http_cmd]: http status code OK 04:12:26 DEBUG --- status code --- 04:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:26 DEBUG --- http response --- 04:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:12:26 DEBUG ----- output ----- 04:12:26 DEBUG { 04:12:26 DEBUG "status": "success", 04:12:26 DEBUG "data": { 04:12:26 DEBUG "resultType": "vector", 04:12:26 DEBUG "result": [ 04:12:26 DEBUG { 04:12:26 DEBUG "metric": { 04:12:26 DEBUG "pod": "ds-idrepo-1" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG [...] 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG "pod": "ds-idrepo-0" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 1713064346, 04:12:26 DEBUG "0" 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG } 04:12:26 INFO [http_cmd]: http status code OK 04:12:26 DEBUG --- status code --- 04:12:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:12:26 DEBUG --- http response --- 04:12:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 04:12:26 DEBUG ----- output ----- 04:12:26 DEBUG { 04:12:26 DEBUG "status": "success", 04:12:26 DEBUG "data": { 04:12:26 DEBUG "resultType": "vector", 04:12:26 DEBUG "result": [ 04:12:26 DEBUG { 04:12:26 DEBUG "metric": { 04:12:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG [...] 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG 04:12:26 DEBUG "pod": "idm-b85f9d978-j5pbm" 04:12:26 DEBUG }, 04:12:26 DEBUG "value": [ 04:12:26 DEBUG 1713064346, 04:12:26 DEBUG "1136.9295491152498" 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG ] 04:12:26 DEBUG } 04:12:26 DEBUG } 04:12:26 INFO 04:12:26 INFO [loop_until]: kubectl --namespace=xlou top pods 04:12:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:12:26 INFO [loop_until]: OK (rc = 0) 04:12:26 DEBUG --- stdout --- 04:12:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 14129m 8975Mi am-7d597cf7d6-8vr8f 11658m 8084Mi am-7d597cf7d6-9wz9h 14844m 8831Mi ds-cts-0 7678m 7574Mi ds-cts-1 8232m 7906Mi ds-cts-2 8207m 7834Mi ds-idrepo-0 12576m 4584Mi ds-idrepo-1 1875m 4301Mi ds-idrepo-2 1777m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 996Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 9m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8078m 2840Mi 04:12:26 DEBUG --- stderr --- 04:12:26 DEBUG 04:12:26 INFO 04:12:26 INFO [loop_until]: kubectl --namespace=xlou top node 04:12:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:12:26 INFO [loop_until]: OK (rc = 0) 04:12:26 DEBUG --- stdout --- 04:12:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4822Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8481m 28% 10021Mi 8% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11430m 38% 11029Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 19973m 66% 3541Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 2012m 6% 6375Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 8193m 27% 11013Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2916Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10838m 36% 9522Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1926m 6% 5520Mi 4% gke-xlou-cdm-ds-0f601751-j488 8814m 29% 9452Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 72m 0% 2216Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 16215m 54% 10389Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13390m 44% 6066Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 7840m 49% 6501Mi 11% 04:12:26 DEBUG --- stderr --- 04:12:26 DEBUG 04:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064405" 04:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064405" 04:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713064405" 04:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064405" 04:13:25 INFO [http_cmd]: http status code OK 04:13:25 DEBUG --- status code --- 04:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:25 DEBUG --- http response --- 04:13:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713064405" 04:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064405" 04:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064405" 04:13:25 ERROR [http_cmd]: ERROR 04:13:25 INFO [http_cmd]: http status code OK 04:13:25 INFO [http_cmd]: http status code OK 04:13:25 ERROR --- status code --- 04:13:25 DEBUG --- status code --- 04:13:25 DEBUG --- status code --- 04:13:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:25 ERROR --- http response --- 04:13:25 DEBUG --- http response --- 04:13:25 DEBUG --- http response --- 04:13:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:13:25 DEBUG ----- output ----- 04:13:25 DEBUG ----- output ----- 04:13:25 DEBUG { 04:13:25 DEBUG { 04:13:25 DEBUG "status": "success", 04:13:25 DEBUG "status": "success", 04:13:25 DEBUG "data": { 04:13:25 DEBUG "data": { 04:13:25 DEBUG "resultType": "vector", 04:13:25 DEBUG "resultType": "vector", 04:13:25 DEBUG "result": [ 04:13:25 DEBUG "result": [ 04:13:25 DEBUG { 04:13:25 DEBUG { 04:13:25 DEBUG "metric": { 04:13:25 DEBUG "metric": { 04:13:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:13:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:13:25 DEBUG }, 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG [...] 04:13:25 DEBUG [...] 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:13:25 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:13:25 DEBUG }, 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 1713064405, 04:13:25 DEBUG 1713064405, 04:13:25 DEBUG "887.5333333333333" 04:13:25 DEBUG "0" 04:13:25 DEBUG ] 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG } 04:13:25 DEBUG ] 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG } 04:13:25 DEBUG } 04:13:25 DEBUG } 04:13:25 INFO [http_cmd]: http status code OK 04:13:25 DEBUG --- status code --- 04:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:25 DEBUG --- http response --- 04:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:25 DEBUG ----- output ----- 04:13:25 DEBUG { 04:13:25 DEBUG "status": "success", 04:13:25 DEBUG "data": { 04:13:25 DEBUG "resultType": "vector", 04:13:25 DEBUG "result": [ 04:13:25 DEBUG { 04:13:25 DEBUG "metric": { 04:13:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG [...] 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 1713064405, 04:13:25 DEBUG "5675438080" 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG } 04:13:25 INFO [http_cmd]: http status code OK 04:13:25 INFO [http_cmd]: http status code OK 04:13:25 DEBUG --- status code --- 04:13:25 DEBUG --- status code --- 04:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:25 DEBUG --- http response --- 04:13:25 DEBUG --- http response --- 04:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:25 DEBUG ----- output ----- 04:13:25 DEBUG { 04:13:25 DEBUG "status": "success", 04:13:25 DEBUG "data": { 04:13:25 DEBUG "resultType": "vector", 04:13:25 DEBUG "result": [ 04:13:25 DEBUG { 04:13:25 DEBUG "metric": { 04:13:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:25 DEBUG 04:13:25 DEBUG ----- output ----- 04:13:25 DEBUG 04:13:25 DEBUG { 04:13:25 DEBUG 04:13:25 DEBUG "status": "success", 04:13:25 DEBUG [...] 04:13:25 DEBUG "data": { 04:13:25 DEBUG 04:13:25 DEBUG "resultType": "vector", 04:13:25 DEBUG 04:13:25 DEBUG "result": [ 04:13:25 DEBUG 04:13:25 DEBUG { 04:13:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:13:25 DEBUG "metric": { 04:13:25 DEBUG }, 04:13:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:13:25 DEBUG "value": [ 04:13:25 DEBUG }, 04:13:25 DEBUG 1713064405, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG "210903" 04:13:25 DEBUG 04:13:25 DEBUG ] 04:13:25 DEBUG 04:13:25 DEBUG } 04:13:25 DEBUG 04:13:25 DEBUG ] 04:13:25 DEBUG [...] 04:13:25 DEBUG } 04:13:25 DEBUG 04:13:25 DEBUG } 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 1713064405, 04:13:25 DEBUG "1343.8666666666666" 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG } 04:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064405" 04:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064405" 04:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064405" 04:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064405" 04:13:25 INFO [http_cmd]: http status code OK 04:13:25 DEBUG --- status code --- 04:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:25 DEBUG --- http response --- 04:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:25 DEBUG ----- output ----- 04:13:25 DEBUG { 04:13:25 DEBUG "status": "success", 04:13:25 DEBUG "data": { 04:13:25 DEBUG "resultType": "vector", 04:13:25 DEBUG "result": [ 04:13:25 DEBUG { 04:13:25 DEBUG "metric": { 04:13:25 DEBUG "pod": "ds-idrepo-1" 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG [...] 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG "pod": "ds-idrepo-0" 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 1713064405, 04:13:25 DEBUG "0" 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG } 04:13:25 INFO [http_cmd]: http status code OK 04:13:25 DEBUG --- status code --- 04:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:25 DEBUG --- http response --- 04:13:25 INFO [http_cmd]: http status code OK 04:13:25 INFO [http_cmd]: http status code OK 04:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:13:25 DEBUG --- status code --- 04:13:25 DEBUG --- status code --- 04:13:25 DEBUG ----- output ----- 04:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:25 DEBUG { 04:13:25 DEBUG --- http response --- 04:13:25 DEBUG --- http response --- 04:13:25 DEBUG "status": "success", 04:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:25 DEBUG "data": { 04:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:25 DEBUG ----- output ----- 04:13:25 DEBUG "resultType": "vector", 04:13:25 DEBUG ----- output ----- 04:13:25 DEBUG { 04:13:25 DEBUG "result": [ 04:13:25 DEBUG { 04:13:25 DEBUG "status": "success", 04:13:25 DEBUG { 04:13:25 DEBUG "status": "success", 04:13:25 DEBUG "data": { 04:13:25 DEBUG "metric": { 04:13:25 DEBUG "data": { 04:13:25 DEBUG "resultType": "vector", 04:13:25 DEBUG "pod": "ds-idrepo-1" 04:13:25 DEBUG "resultType": "vector", 04:13:25 DEBUG "result": [ 04:13:25 DEBUG }, 04:13:25 DEBUG "result": [ 04:13:25 DEBUG { 04:13:25 DEBUG "value": [ 04:13:25 DEBUG { 04:13:25 DEBUG "metric": { 04:13:25 DEBUG 04:13:25 DEBUG "metric": { 04:13:25 DEBUG "instance": "10.142.0.107:9100" 04:13:25 DEBUG 04:13:25 DEBUG "instance": "10.142.0.107:9100" 04:13:25 DEBUG }, 04:13:25 DEBUG 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG [...] 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG [...] 04:13:25 DEBUG "pod": "ds-idrepo-0" 04:13:25 DEBUG [...] 04:13:25 DEBUG 04:13:25 DEBUG }, 04:13:25 DEBUG 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 1713064405, 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG "0" 04:13:25 DEBUG "instance": "10.142.0.43:9100" 04:13:25 DEBUG "instance": "10.142.0.43:9100" 04:13:25 DEBUG ] 04:13:25 DEBUG }, 04:13:25 DEBUG }, 04:13:25 DEBUG } 04:13:25 DEBUG "value": [ 04:13:25 DEBUG "value": [ 04:13:25 DEBUG ] 04:13:25 DEBUG 1713064405, 04:13:25 DEBUG 1713064405, 04:13:25 DEBUG } 04:13:25 DEBUG "2999.7" 04:13:25 DEBUG "12693107712" 04:13:25 DEBUG } 04:13:25 DEBUG ] 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG } 04:13:25 DEBUG ] 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG } 04:13:25 DEBUG } 04:13:25 DEBUG } 04:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713064405" 04:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064405" 04:13:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064405" 04:13:25 INFO [http_cmd]: http status code OK 04:13:25 DEBUG --- status code --- 04:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:25 DEBUG --- http response --- 04:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:25 DEBUG ----- output ----- 04:13:25 DEBUG { 04:13:25 DEBUG "status": "success", 04:13:25 DEBUG "data": { 04:13:25 DEBUG "resultType": "vector", 04:13:25 DEBUG "result": [ 04:13:25 DEBUG { 04:13:25 DEBUG "metric": { 04:13:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG [...] 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 1713064405, 04:13:25 DEBUG "0.015" 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG } 04:13:25 INFO [http_cmd]: http status code OK 04:13:25 DEBUG --- status code --- 04:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:25 DEBUG --- http response --- 04:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:25 DEBUG ----- output ----- 04:13:25 DEBUG { 04:13:25 DEBUG "status": "success", 04:13:25 DEBUG "data": { 04:13:25 DEBUG "resultType": "vector", 04:13:25 DEBUG "result": [ 04:13:25 DEBUG { 04:13:25 DEBUG "metric": { 04:13:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG [...] 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 1713064405, 04:13:25 DEBUG "1345.0333333333333" 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG } 04:13:25 INFO [http_cmd]: http status code OK 04:13:25 DEBUG --- status code --- 04:13:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:25 DEBUG --- http response --- 04:13:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:25 DEBUG ----- output ----- 04:13:25 DEBUG { 04:13:25 DEBUG "status": "success", 04:13:25 DEBUG "data": { 04:13:25 DEBUG "resultType": "vector", 04:13:25 DEBUG "result": [ 04:13:25 DEBUG { 04:13:25 DEBUG "metric": { 04:13:25 DEBUG "pod": "ds-idrepo-1" 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG [...] 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG 04:13:25 DEBUG "pod": "ds-idrepo-0" 04:13:25 DEBUG }, 04:13:25 DEBUG "value": [ 04:13:25 DEBUG 1713064405, 04:13:25 DEBUG "0" 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG ] 04:13:25 DEBUG } 04:13:25 DEBUG } 04:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064406" 04:13:26 INFO [http_cmd]: http status code OK 04:13:26 DEBUG --- status code --- 04:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:26 DEBUG --- http response --- 04:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:26 DEBUG ----- output ----- 04:13:26 DEBUG { 04:13:26 DEBUG "status": "success", 04:13:26 DEBUG "data": { 04:13:26 DEBUG "resultType": "vector", 04:13:26 DEBUG "result": [ 04:13:26 DEBUG { 04:13:26 DEBUG "metric": { 04:13:26 DEBUG "instance": "10.142.0.107:9100" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG [...] 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG "instance": "10.142.0.43:9100" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 1713064406, 04:13:26 DEBUG "0.0004333333333332708" 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG } 04:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064406" 04:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064406" 04:13:26 INFO [http_cmd]: http status code OK 04:13:26 INFO [http_cmd]: http status code OK 04:13:26 DEBUG --- status code --- 04:13:26 DEBUG --- status code --- 04:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:26 DEBUG --- http response --- 04:13:26 DEBUG --- http response --- 04:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:26 DEBUG ----- output ----- 04:13:26 DEBUG { 04:13:26 DEBUG "status": "success", 04:13:26 DEBUG "data": { 04:13:26 DEBUG "resultType": "vector", 04:13:26 DEBUG "result": [ 04:13:26 DEBUG { 04:13:26 DEBUG "metric": { 04:13:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG [...] 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:26 DEBUG }, 04:13:26 DEBUG ----- output ----- 04:13:26 DEBUG "value": [ 04:13:26 DEBUG { 04:13:26 DEBUG 1713064406, 04:13:26 DEBUG "status": "success", 04:13:26 DEBUG "0" 04:13:26 DEBUG "data": { 04:13:26 DEBUG ] 04:13:26 DEBUG "resultType": "vector", 04:13:26 DEBUG } 04:13:26 DEBUG "result": [ 04:13:26 DEBUG ] 04:13:26 DEBUG { 04:13:26 DEBUG } 04:13:26 DEBUG "metric": { 04:13:26 DEBUG } 04:13:26 DEBUG "pod": "ds-idrepo-1" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG [...] 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG "pod": "ds-idrepo-0" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 1713064406, 04:13:26 DEBUG "0" 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG } 04:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064406" 04:13:26 INFO [http_cmd]: http status code OK 04:13:26 DEBUG --- status code --- 04:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:26 DEBUG --- http response --- 04:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:13:26 DEBUG ----- output ----- 04:13:26 DEBUG { 04:13:26 DEBUG "status": "success", 04:13:26 DEBUG "data": { 04:13:26 DEBUG "resultType": "vector", 04:13:26 DEBUG "result": [ 04:13:26 DEBUG { 04:13:26 DEBUG "metric": { 04:13:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG [...] 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 1713064406, 04:13:26 DEBUG "4.213053328172464" 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG } 04:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713064406" 04:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064406" 04:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064406" 04:13:26 INFO [http_cmd]: http status code OK 04:13:26 DEBUG --- status code --- 04:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:26 DEBUG --- http response --- 04:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:13:26 DEBUG ----- output ----- 04:13:26 DEBUG { 04:13:26 DEBUG "status": "success", 04:13:26 DEBUG "data": { 04:13:26 DEBUG "resultType": "vector", 04:13:26 DEBUG "result": [ 04:13:26 DEBUG { 04:13:26 DEBUG "metric": { 04:13:26 DEBUG "pod": "ds-idrepo-1" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG [...] 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG "pod": "ds-cts-1" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 1713064406, 04:13:26 DEBUG "0" 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG } 04:13:26 INFO [http_cmd]: http status code OK 04:13:26 DEBUG --- status code --- 04:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:26 DEBUG --- http response --- 04:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:13:26 DEBUG ----- output ----- 04:13:26 DEBUG { 04:13:26 DEBUG "status": "success", 04:13:26 DEBUG "data": { 04:13:26 DEBUG "resultType": "vector", 04:13:26 DEBUG "result": [ 04:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064406" 04:13:26 DEBUG { 04:13:26 DEBUG "metric": { 04:13:26 DEBUG "pod": "ds-idrepo-1" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG [...] 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG "pod": "ds-cts-1" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 1713064406, 04:13:26 DEBUG "0" 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG } 04:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064406" 04:13:26 INFO [http_cmd]: http status code OK 04:13:26 DEBUG --- status code --- 04:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:26 DEBUG --- http response --- 04:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:13:26 DEBUG ----- output ----- 04:13:26 DEBUG { 04:13:26 DEBUG "status": "success", 04:13:26 DEBUG "data": { 04:13:26 DEBUG "resultType": "vector", 04:13:26 DEBUG "result": [ 04:13:26 DEBUG { 04:13:26 DEBUG "metric": { 04:13:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG [...] 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 1713064406, 04:13:26 DEBUG "1343.9333333333334" 04:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713064406" 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG } 04:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064406" 04:13:26 INFO [http_cmd]: http status code OK 04:13:26 INFO [http_cmd]: http status code OK 04:13:26 DEBUG --- status code --- 04:13:26 DEBUG --- status code --- 04:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:26 DEBUG --- http response --- 04:13:26 DEBUG --- http response --- 04:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:26 DEBUG ----- output ----- 04:13:26 DEBUG { 04:13:26 DEBUG "status": "success", 04:13:26 DEBUG "data": { 04:13:26 DEBUG "resultType": "vector", 04:13:26 DEBUG "result": [ 04:13:26 DEBUG { 04:13:26 DEBUG "metric": { 04:13:26 DEBUG "instance": "10.142.0.107:9100" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG [...] 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG "instance": "10.142.0.43:9100" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 1713064406, 04:13:26 DEBUG "5522.400000000001" 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG } 04:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:13:26 DEBUG ----- output ----- 04:13:26 DEBUG { 04:13:26 DEBUG "status": "success", 04:13:26 DEBUG "data": { 04:13:26 DEBUG "resultType": "vector", 04:13:26 DEBUG "result": [ 04:13:26 DEBUG { 04:13:26 DEBUG "metric": { 04:13:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG [...] 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 1713064406, 04:13:26 DEBUG "33262712.072415438" 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG } 04:13:26 INFO [http_cmd]: http status code OK 04:13:26 DEBUG --- status code --- 04:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:26 DEBUG --- http response --- 04:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:26 DEBUG ----- output ----- 04:13:26 DEBUG { 04:13:26 DEBUG "status": "success", 04:13:26 DEBUG "data": { 04:13:26 DEBUG "resultType": "vector", 04:13:26 DEBUG "result": [ 04:13:26 DEBUG { 04:13:26 DEBUG "metric": { 04:13:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG [...] 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 1713064406, 04:13:26 DEBUG "8.044165170655985" 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG } 04:13:26 INFO [http_cmd]: http status code OK 04:13:26 DEBUG --- status code --- 04:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:26 DEBUG --- http response --- 04:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:13:26 DEBUG ----- output ----- 04:13:26 DEBUG { 04:13:26 DEBUG "status": "success", 04:13:26 DEBUG "data": { 04:13:26 DEBUG "resultType": "vector", 04:13:26 DEBUG "result": [ 04:13:26 DEBUG { 04:13:26 DEBUG "metric": { 04:13:26 DEBUG "instance": "10.142.0.107:9100" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG [...] 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG "instance": "10.142.0.43:9100" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 1713064406, 04:13:26 DEBUG "0" 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG } 04:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064406" 04:13:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064406" 04:13:26 INFO [http_cmd]: http status code OK 04:13:26 INFO [http_cmd]: http status code OK 04:13:26 DEBUG --- status code --- 04:13:26 DEBUG --- status code --- 04:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:13:26 DEBUG --- http response --- 04:13:26 DEBUG --- http response --- 04:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:13:26 DEBUG ----- output ----- 04:13:26 DEBUG { 04:13:26 DEBUG "status": "success", 04:13:26 DEBUG "data": { 04:13:26 DEBUG "resultType": "vector", 04:13:26 DEBUG "result": [ 04:13:26 DEBUG { 04:13:26 DEBUG "metric": { 04:13:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG [...] 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 1713064406, 04:13:26 DEBUG "0" 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG } 04:13:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:13:26 DEBUG ----- output ----- 04:13:26 DEBUG { 04:13:26 DEBUG "status": "success", 04:13:26 DEBUG "data": { 04:13:26 DEBUG "resultType": "vector", 04:13:26 DEBUG "result": [ 04:13:26 DEBUG { 04:13:26 DEBUG "metric": { 04:13:26 DEBUG "pod": "ds-idrepo-1" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG [...] 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG 04:13:26 DEBUG "pod": "ds-idrepo-0" 04:13:26 DEBUG }, 04:13:26 DEBUG "value": [ 04:13:26 DEBUG 1713064406, 04:13:26 DEBUG "0" 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG ] 04:13:26 DEBUG } 04:13:26 DEBUG } 04:13:26 INFO 04:13:26 INFO [loop_until]: kubectl --namespace=xlou top pods 04:13:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:13:26 INFO [loop_until]: OK (rc = 0) 04:13:26 DEBUG --- stdout --- 04:13:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 7189m 8431Mi am-7d597cf7d6-8vr8f 6458m 8083Mi am-7d597cf7d6-9wz9h 8554m 8817Mi ds-cts-0 5603m 7595Mi ds-cts-1 6070m 7916Mi ds-cts-2 5830m 7842Mi ds-idrepo-0 7654m 4584Mi ds-idrepo-1 1191m 4301Mi ds-idrepo-2 1103m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 996Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 8m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7998m 4135Mi 04:13:26 DEBUG --- stderr --- 04:13:26 DEBUG 04:13:26 INFO 04:13:26 INFO [loop_until]: kubectl --namespace=xlou top node 04:13:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:13:26 INFO [loop_until]: OK (rc = 0) 04:13:26 DEBUG --- stdout --- 04:13:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4830Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5905m 19% 10022Mi 8% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7201m 24% 10492Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11457m 38% 3617Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1236m 4% 6370Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 5832m 19% 11028Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2914Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 6953m 23% 9521Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1173m 3% 5532Mi 4% gke-xlou-cdm-ds-0f601751-j488 6206m 20% 9449Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 70m 0% 2229Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 9501m 31% 10371Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 7622m 25% 6061Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8192m 51% 8480Mi 14% 04:13:26 DEBUG --- stderr --- 04:13:26 DEBUG 04:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713064465" 04:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064465" 04:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064465" 04:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064465" 04:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064465" 04:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064465" 04:14:25 INFO [http_cmd]: http status code OK 04:14:25 DEBUG --- status code --- 04:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:25 DEBUG --- http response --- 04:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:25 DEBUG ----- output ----- 04:14:25 DEBUG { 04:14:25 DEBUG "status": "success", 04:14:25 DEBUG "data": { 04:14:25 DEBUG "resultType": "vector", 04:14:25 DEBUG "result": [ 04:14:25 DEBUG { 04:14:25 DEBUG "metric": { 04:14:25 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:14:25 DEBUG }, 04:14:25 DEBUG "value": [ 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG [...] 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:14:25 DEBUG }, 04:14:25 DEBUG "value": [ 04:14:25 DEBUG 1713064465, 04:14:25 DEBUG "7305908224" 04:14:25 DEBUG ] 04:14:25 DEBUG } 04:14:25 DEBUG ] 04:14:25 DEBUG } 04:14:25 DEBUG } 04:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713064465" 04:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064465" 04:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064465" 04:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064465" 04:14:25 INFO [http_cmd]: http status code OK 04:14:25 DEBUG --- status code --- 04:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:25 DEBUG --- http response --- 04:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:25 DEBUG ----- output ----- 04:14:25 DEBUG { 04:14:25 DEBUG "status": "success", 04:14:25 DEBUG "data": { 04:14:25 DEBUG "resultType": "vector", 04:14:25 DEBUG "result": [ 04:14:25 DEBUG { 04:14:25 DEBUG "metric": { 04:14:25 DEBUG "pod": "ds-idrepo-1" 04:14:25 DEBUG }, 04:14:25 DEBUG "value": [ 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG [...] 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG "pod": "ds-idrepo-0" 04:14:25 DEBUG }, 04:14:25 DEBUG "value": [ 04:14:25 DEBUG 1713064465, 04:14:25 DEBUG "0" 04:14:25 DEBUG ] 04:14:25 DEBUG } 04:14:25 DEBUG ] 04:14:25 DEBUG } 04:14:25 DEBUG } 04:14:25 INFO [http_cmd]: http status code OK 04:14:25 INFO [http_cmd]: http status code OK 04:14:25 INFO [http_cmd]: http status code OK 04:14:25 DEBUG --- status code --- 04:14:25 DEBUG --- status code --- 04:14:25 DEBUG --- status code --- 04:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:25 DEBUG --- http response --- 04:14:25 DEBUG --- http response --- 04:14:25 DEBUG --- http response --- 04:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:25 DEBUG ----- output ----- 04:14:25 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:14:25 DEBUG ----- output ----- 04:14:25 DEBUG { 04:14:25 DEBUG { 04:14:25 INFO [http_cmd]: http status code OK 04:14:25 DEBUG "status": "success", 04:14:25 DEBUG "status": "success", 04:14:25 DEBUG --- status code --- 04:14:25 DEBUG "data": { 04:14:25 DEBUG "data": { 04:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:25 DEBUG "resultType": "vector", 04:14:25 DEBUG "resultType": "vector", 04:14:25 DEBUG --- http response --- 04:14:25 DEBUG "result": [ 04:14:25 DEBUG "result": [ 04:14:25 DEBUG { 04:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 04:14:25 DEBUG { 04:14:25 INFO [http_cmd]: http status code OK 04:14:25 DEBUG "metric": { 04:14:25 INFO [http_cmd]: http status code OK 04:14:25 ERROR [http_cmd]: ERROR 04:14:25 DEBUG ----- output ----- 04:14:25 DEBUG "metric": { 04:14:25 DEBUG --- status code --- 04:14:25 DEBUG "pod": "ds-idrepo-1" 04:14:25 DEBUG --- status code --- 04:14:25 ERROR --- status code --- 04:14:25 DEBUG { 04:14:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:25 DEBUG }, 04:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:25 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:14:25 DEBUG "status": "success", 04:14:25 DEBUG }, 04:14:25 DEBUG --- http response --- 04:14:25 DEBUG "value": [ 04:14:25 DEBUG 04:14:25 ERROR --- http response --- 04:14:25 DEBUG "data": { 04:14:25 DEBUG "value": [ 04:14:25 INFO [http_cmd]: http status code OK 04:14:25 DEBUG --- http response --- 04:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064465" 04:14:25 DEBUG 04:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:25 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:14:25 DEBUG "resultType": "vector", 04:14:25 DEBUG 04:14:25 DEBUG --- status code --- 04:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:25 DEBUG 04:14:25 DEBUG ----- output ----- 04:14:25 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:14:25 DEBUG "result": [ 04:14:25 DEBUG 04:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:25 DEBUG ----- output ----- 04:14:25 DEBUG [...] 04:14:25 DEBUG { 04:14:25 DEBUG { 04:14:25 DEBUG 04:14:25 DEBUG --- http response --- 04:14:25 DEBUG { 04:14:25 DEBUG 04:14:25 DEBUG "status": "success", 04:14:25 DEBUG "metric": { 04:14:25 DEBUG [...] 04:14:25 DEBUG "status": "success", 04:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:25 DEBUG 04:14:25 DEBUG "data": { 04:14:25 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:14:25 DEBUG 04:14:25 DEBUG "data": { 04:14:25 DEBUG ----- output ----- 04:14:25 DEBUG 04:14:25 DEBUG "resultType": "vector", 04:14:25 DEBUG }, 04:14:25 DEBUG 04:14:25 DEBUG "resultType": "vector", 04:14:25 DEBUG { 04:14:25 DEBUG "pod": "ds-idrepo-0" 04:14:25 DEBUG "result": [ 04:14:25 DEBUG "value": [ 04:14:25 DEBUG 04:14:25 DEBUG "result": [ 04:14:25 DEBUG "status": "success", 04:14:25 DEBUG }, 04:14:25 DEBUG { 04:14:25 DEBUG 04:14:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:14:25 DEBUG { 04:14:25 DEBUG "data": { 04:14:25 DEBUG "value": [ 04:14:25 DEBUG "metric": { 04:14:25 DEBUG 04:14:25 DEBUG }, 04:14:25 DEBUG "metric": { 04:14:25 DEBUG "resultType": "vector", 04:14:25 DEBUG 1713064465, 04:14:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:14:25 DEBUG 04:14:25 DEBUG "value": [ 04:14:25 DEBUG "instance": "10.142.0.107:9100" 04:14:25 DEBUG "result": [ 04:14:25 DEBUG "0" 04:14:25 DEBUG }, 04:14:25 DEBUG [...] 04:14:25 DEBUG 1713064465, 04:14:25 DEBUG }, 04:14:25 DEBUG { 04:14:25 DEBUG ] 04:14:25 DEBUG "value": [ 04:14:25 DEBUG 04:14:25 DEBUG "263742.5" 04:14:25 DEBUG "value": [ 04:14:25 DEBUG "metric": { 04:14:25 DEBUG } 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG ] 04:14:25 DEBUG 04:14:25 DEBUG "instance": "10.142.0.107:9100" 04:14:25 DEBUG ] 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG } 04:14:25 DEBUG 04:14:25 DEBUG }, 04:14:25 DEBUG } 04:14:25 DEBUG 04:14:25 DEBUG "pod": "ds-idrepo-2" 04:14:25 DEBUG ] 04:14:25 DEBUG 04:14:25 DEBUG "value": [ 04:14:25 DEBUG } 04:14:25 DEBUG [...] 04:14:25 DEBUG }, 04:14:25 DEBUG } 04:14:25 DEBUG [...] 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG "value": [ 04:14:25 DEBUG } 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG 1713064465, 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG "0" 04:14:25 DEBUG 04:14:25 DEBUG [...] 04:14:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:14:25 DEBUG ] 04:14:25 DEBUG "instance": "10.142.0.43:9100" 04:14:25 DEBUG 04:14:25 DEBUG }, 04:14:25 DEBUG } 04:14:25 DEBUG }, 04:14:25 DEBUG 04:14:25 DEBUG "value": [ 04:14:25 DEBUG ] 04:14:25 DEBUG "value": [ 04:14:25 DEBUG 04:14:25 DEBUG 1713064465, 04:14:25 DEBUG } 04:14:25 DEBUG 1713064465, 04:14:25 DEBUG "instance": "10.142.0.43:9100" 04:14:25 DEBUG "802.8333333333334" 04:14:25 DEBUG } 04:14:25 DEBUG "4538.866666666666" 04:14:25 DEBUG }, 04:14:25 DEBUG ] 04:14:25 DEBUG ] 04:14:25 DEBUG "value": [ 04:14:25 DEBUG } 04:14:25 DEBUG } 04:14:25 DEBUG 1713064465, 04:14:25 DEBUG ] 04:14:25 DEBUG ] 04:14:25 DEBUG "12693789696" 04:14:25 DEBUG } 04:14:25 DEBUG } 04:14:25 DEBUG ] 04:14:25 DEBUG } 04:14:25 DEBUG } 04:14:25 DEBUG } 04:14:25 DEBUG ] 04:14:25 DEBUG } 04:14:25 DEBUG } 04:14:25 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713064465" 04:14:25 INFO [http_cmd]: http status code OK 04:14:25 DEBUG --- status code --- 04:14:25 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:25 DEBUG --- http response --- 04:14:25 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:25 DEBUG ----- output ----- 04:14:25 DEBUG { 04:14:25 DEBUG "status": "success", 04:14:25 DEBUG "data": { 04:14:25 DEBUG "resultType": "vector", 04:14:25 DEBUG "result": [ 04:14:25 DEBUG { 04:14:25 DEBUG "metric": { 04:14:25 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:14:25 DEBUG }, 04:14:25 DEBUG "value": [ 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG [...] 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG 04:14:25 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:14:25 DEBUG }, 04:14:25 DEBUG "value": [ 04:14:25 DEBUG 1713064465, 04:14:25 DEBUG "1319.7666666666667" 04:14:25 DEBUG ] 04:14:25 DEBUG } 04:14:25 DEBUG ] 04:14:25 DEBUG } 04:14:25 DEBUG } 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064465, 04:14:26 DEBUG "0.015" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064466" 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "pod": "ds-idrepo-1" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "pod": "ds-idrepo-0" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "0" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064466" 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "1320.5666666666666" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064466" 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "instance": "10.142.0.107:9100" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "instance": "10.142.0.43:9100" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "0.00030000000000001136" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064466" 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064466" 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "0" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:26 DEBUG ] 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG } 04:14:26 DEBUG { 04:14:26 DEBUG } 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "pod": "ds-idrepo-1" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "pod": "ds-idrepo-0" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "0" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064466" 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "3.4893863209648663" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713064466" 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064466" 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064466" 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064466" 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "pod": "ds-idrepo-1" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "pod": "ds-cts-1" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "0" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "pod": "ds-idrepo-1" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "pod": "ds-cts-1" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "0.0001683888888888889" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064466" 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG --- http response --- 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064466" 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "1319.6666666666667" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713064466" 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "instance": "10.142.0.107:9100" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "instance": "10.142.0.43:9100" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "0.00033333333333333365" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "instance": "10.142.0.107:9100" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "instance": "10.142.0.43:9100" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "5691.933333333333" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "9118421.19506309" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "8.029014269947718" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064466" 04:14:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064466" 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "pod": "ds-idrepo-1" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "pod": "ds-idrepo-0" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "0" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 INFO [http_cmd]: http status code OK 04:14:26 DEBUG --- status code --- 04:14:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:14:26 DEBUG --- http response --- 04:14:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:14:26 DEBUG ----- output ----- 04:14:26 DEBUG { 04:14:26 DEBUG "status": "success", 04:14:26 DEBUG "data": { 04:14:26 DEBUG "resultType": "vector", 04:14:26 DEBUG "result": [ 04:14:26 DEBUG { 04:14:26 DEBUG "metric": { 04:14:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG [...] 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG 04:14:26 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:14:26 DEBUG }, 04:14:26 DEBUG "value": [ 04:14:26 DEBUG 1713064466, 04:14:26 DEBUG "0" 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG ] 04:14:26 DEBUG } 04:14:26 DEBUG } 04:14:26 INFO 04:14:26 INFO [loop_until]: kubectl --namespace=xlou top pods 04:14:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:14:26 INFO [loop_until]: OK (rc = 0) 04:14:26 DEBUG --- stdout --- 04:14:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5446m 8431Mi am-7d597cf7d6-8vr8f 5531m 8082Mi am-7d597cf7d6-9wz9h 6930m 8813Mi ds-cts-0 2697m 7603Mi ds-cts-1 2731m 7944Mi ds-cts-2 2749m 7839Mi ds-idrepo-0 6541m 4584Mi ds-idrepo-1 1027m 4300Mi ds-idrepo-2 967m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 5m 996Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 7m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8150m 6877Mi 04:14:26 DEBUG --- stderr --- 04:14:26 DEBUG 04:14:26 INFO 04:14:26 INFO [loop_until]: kubectl --namespace=xlou top node 04:14:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:14:26 INFO [loop_until]: OK (rc = 0) 04:14:26 DEBUG --- stdout --- 04:14:26 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 4833Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2784m 9% 10027Mi 8% gke-xlou-cdm-default-pool-1b0769c8-nhlf 5906m 19% 10487Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11455m 38% 3618Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1162m 3% 6392Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 2672m 8% 11034Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2912Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 5807m 19% 9524Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1077m 3% 5527Mi 4% gke-xlou-cdm-ds-0f601751-j488 2821m 9% 9470Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 72m 0% 2208Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 7507m 25% 10365Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 6619m 22% 6069Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8241m 51% 10207Mi 17% 04:14:26 DEBUG --- stderr --- 04:14:26 DEBUG 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG --- http response --- 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713064526" 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG { 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "data": { 04:15:26 DEBUG "data": { 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:26 DEBUG "resultType": "vector", 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG "result": [ 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG "result": [ 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG { 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG { 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "metric": { 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "data": { 04:15:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:15:26 DEBUG "instance": "10.142.0.107:9100" 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:26 DEBUG "instance": "10.142.0.107:9100" 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG }, 04:15:26 DEBUG }, 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG }, 04:15:26 DEBUG "result": [ 04:15:26 DEBUG "value": [ 04:15:26 DEBUG "value": [ 04:15:26 DEBUG { 04:15:26 DEBUG "value": [ 04:15:26 DEBUG { 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG 04:15:26 DEBUG "metric": { 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "data": { 04:15:26 DEBUG 04:15:26 DEBUG "pod": "ds-idrepo-1" 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG 04:15:26 DEBUG }, 04:15:26 DEBUG [...] 04:15:26 DEBUG [...] 04:15:26 DEBUG "result": [ 04:15:26 DEBUG [...] 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG { 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "metric": { 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:15:26 DEBUG "instance": "10.142.0.43:9100" 04:15:26 DEBUG }, 04:15:26 DEBUG "instance": "10.142.0.43:9100" 04:15:26 DEBUG [...] 04:15:26 DEBUG }, 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG }, 04:15:26 DEBUG 04:15:26 DEBUG "value": [ 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG 04:15:26 DEBUG "0.015" 04:15:26 DEBUG "12694330368" 04:15:26 DEBUG 04:15:26 DEBUG "4449.633333333333" 04:15:26 DEBUG "pod": "ds-idrepo-0" 04:15:26 DEBUG ] 04:15:26 DEBUG ] 04:15:26 DEBUG [...] 04:15:26 DEBUG ] 04:15:26 DEBUG }, 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 DEBUG 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG } 04:15:26 DEBUG "value": [ 04:15:26 DEBUG ] 04:15:26 DEBUG ] 04:15:26 DEBUG 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG ] 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 DEBUG 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG } 04:15:26 DEBUG "0" 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG }, 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:26 DEBUG } 04:15:26 DEBUG "value": [ 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG ] 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG { 04:15:26 DEBUG } 04:15:26 DEBUG "426.3333333333333" 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG "data": { 04:15:26 DEBUG } 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG ] 04:15:26 DEBUG "result": [ 04:15:26 DEBUG } 04:15:26 DEBUG { 04:15:26 DEBUG } 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "223166" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713064526" 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "8624205824" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 ERROR [http_cmd]: ERROR 04:15:26 ERROR --- status code --- 04:15:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:15:26 ERROR --- http response --- 04:15:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:15:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "1379.3666666666666" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "pod": "ds-idrepo-1" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "ds-idrepo-0" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "0" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "ds-idrepo-1" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "0" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "426.3333333333333" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "pod": "ds-idrepo-1" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "ds-idrepo-0" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "0" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064526" 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "instance": "10.142.0.107:9100" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "instance": "10.142.0.43:9100" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "0.00030000000000001136" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "pod": "ds-idrepo-1" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "ds-idrepo-0" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "0" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "0" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "ds-idrepo-1" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "0.7664043242514037" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064526" 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "pod": "ds-idrepo-1" 04:15:26 DEBUG "pod": "ds-idrepo-1" 04:15:26 DEBUG }, 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "ds-cts-1" 04:15:26 DEBUG "pod": "ds-cts-1" 04:15:26 DEBUG }, 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "0" 04:15:26 DEBUG "0.0002" 04:15:26 DEBUG ] 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713064526" 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "426.3333333333333" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "ds-idrepo-1" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "726.7411678202885" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "instance": "10.142.0.107:9100" 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG }, 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG "value": [ 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "instance": "10.142.0.43:9100" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "0" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "instance": "10.142.0.107:9100" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "instance": "10.142.0.43:9100" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "5547" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "8.020881653669559" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064526" 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "ds-idrepo-1" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "3737.5260059329125" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO [http_cmd]: http status code OK 04:15:26 DEBUG --- status code --- 04:15:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:15:26 DEBUG --- http response --- 04:15:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:15:26 DEBUG ----- output ----- 04:15:26 DEBUG { 04:15:26 DEBUG "status": "success", 04:15:26 DEBUG "data": { 04:15:26 DEBUG "resultType": "vector", 04:15:26 DEBUG "result": [ 04:15:26 DEBUG { 04:15:26 DEBUG "metric": { 04:15:26 DEBUG "pod": "ds-idrepo-1" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG [...] 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG 04:15:26 DEBUG "pod": "ds-idrepo-0" 04:15:26 DEBUG }, 04:15:26 DEBUG "value": [ 04:15:26 DEBUG 1713064526, 04:15:26 DEBUG "0" 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG ] 04:15:26 DEBUG } 04:15:26 DEBUG } 04:15:26 INFO 04:15:26 INFO [loop_until]: kubectl --namespace=xlou top pods 04:15:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:15:26 INFO [loop_until]: OK (rc = 0) 04:15:26 DEBUG --- stdout --- 04:15:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 7880m 8438Mi am-7d597cf7d6-8vr8f 7370m 8086Mi am-7d597cf7d6-9wz9h 9689m 8821Mi ds-cts-0 8185m 7592Mi ds-cts-1 8914m 7949Mi ds-cts-2 8680m 7846Mi ds-idrepo-0 7668m 4584Mi ds-idrepo-1 1202m 4301Mi ds-idrepo-2 1109m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 996Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 6m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7946m 7842Mi 04:15:26 DEBUG --- stderr --- 04:15:26 DEBUG 04:15:27 INFO 04:15:27 INFO [loop_until]: kubectl --namespace=xlou top node 04:15:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:15:27 INFO [loop_until]: OK (rc = 0) 04:15:27 DEBUG --- stdout --- 04:15:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4830Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8672m 29% 10019Mi 8% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7797m 26% 10485Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10246m 34% 3627Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1228m 4% 6393Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 8532m 28% 11032Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2910Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 7732m 25% 9525Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1191m 3% 5516Mi 4% gke-xlou-cdm-ds-0f601751-j488 9003m 30% 9465Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 2216Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 10395m 34% 10385Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 7942m 26% 6071Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8267m 52% 10928Mi 18% 04:15:27 DEBUG --- stderr --- 04:15:27 DEBUG 04:16:25 WARNING File /tmp/lodemon_data-20240414_035725/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [14/Apr/2024 04:16:25] "GET /monitoring/average?start_time=20240414_031236&stop_time=20240414_031424 HTTP/1.1" 200 - 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064586" 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064586" 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713064586" 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:16:26 DEBUG --- http response --- 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG { 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG { 04:16:26 DEBUG { 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:26 DEBUG "data": { 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "data": { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "data": { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "pod": "ds-idrepo-1" 04:16:26 DEBUG }, 04:16:26 DEBUG "result": [ 04:16:26 DEBUG }, 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713064586" 04:16:26 DEBUG "result": [ 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG { 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG "value": [ 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG { 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG "metric": { 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG "metric": { 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG "instance": "10.142.0.107:9100" 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG }, 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG }, 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG "value": [ 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG 04:16:26 DEBUG "value": [ 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG { 04:16:26 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:16:26 DEBUG 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG }, 04:16:26 DEBUG 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG 04:16:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:16:26 DEBUG [...] 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "data": { 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "pod": "ds-idrepo-0" 04:16:26 DEBUG }, 04:16:26 DEBUG 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "resultType": "vector", 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG [...] 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG "result": [ 04:16:26 DEBUG "190900573.44611973" 04:16:26 DEBUG 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG }, 04:16:26 DEBUG { 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG { 04:16:26 DEBUG ] 04:16:26 DEBUG 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "0.06666666666666667" 04:16:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 ERROR [http_cmd]: ERROR 04:16:26 DEBUG "metric": { 04:16:26 DEBUG } 04:16:26 DEBUG 04:16:26 DEBUG "value": [ 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:16:26 DEBUG ] 04:16:26 DEBUG }, 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:16:26 ERROR --- status code --- 04:16:26 DEBUG "instance": "10.142.0.107:9100" 04:16:26 DEBUG ] 04:16:26 DEBUG "instance": "10.142.0.43:9100" 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:16:26 DEBUG }, 04:16:26 DEBUG } 04:16:26 DEBUG "value": [ 04:16:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:16:26 DEBUG }, 04:16:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:16:26 DEBUG }, 04:16:26 DEBUG } 04:16:26 DEBUG }, 04:16:26 DEBUG "0" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG ] 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 ERROR --- http response --- 04:16:26 DEBUG "value": [ 04:16:26 DEBUG } 04:16:26 DEBUG "value": [ 04:16:26 DEBUG ] 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG } 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG } 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "1348.2" 04:16:26 DEBUG } 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:16:26 DEBUG "3006.9666666666667" 04:16:26 DEBUG ] 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG ] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG } 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG [...] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG ] 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG ] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG } 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG } 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG } 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG } 04:16:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:16:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:16:26 DEBUG 04:16:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:16:26 DEBUG "instance": "10.142.0.43:9100" 04:16:26 DEBUG }, 04:16:26 DEBUG }, 04:16:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:16:26 DEBUG }, 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG "value": [ 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "0.015" 04:16:26 DEBUG "148642" 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "0.06666666666666667" 04:16:26 DEBUG "12694991872" 04:16:26 DEBUG ] 04:16:26 DEBUG ] 04:16:26 DEBUG "8685727744" 04:16:26 DEBUG ] 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG } 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "pod": "ds-idrepo-1" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "pod": "ds-idrepo-0" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "0" 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "pod": "ds-idrepo-1" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "pod": "ds-idrepo-0" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "0" 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064586" 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "0" 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "instance": "10.142.0.107:9100" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "instance": "10.142.0.43:9100" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "0.0002333333333332632" 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "pod": "ds-idrepo-1" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "pod": "ds-idrepo-0" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "0" 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064586" 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "instance": "10.142.0.107:9100" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "instance": "10.142.0.43:9100" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "5563.5666666666675" 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "1.364307532768578" 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "0.06666666666666667" 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064586" 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713064586" 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "pod": "ds-idrepo-1" 04:16:26 DEBUG "pod": "ds-idrepo-1" 04:16:26 DEBUG }, 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "pod": "ds-cts-1" 04:16:26 DEBUG "pod": "ds-cts-1" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG }, 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG "0" 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG ] 04:16:26 DEBUG "0.00006666666666666667" 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "1321.7827981537546" 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "instance": "10.142.0.107:9100" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "instance": "10.142.0.43:9100" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "0" 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "2.5481276935256103" 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "0" 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064586" 04:16:26 INFO [http_cmd]: http status code OK 04:16:26 DEBUG --- status code --- 04:16:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:16:26 DEBUG --- http response --- 04:16:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:16:26 DEBUG ----- output ----- 04:16:26 DEBUG { 04:16:26 DEBUG "status": "success", 04:16:26 DEBUG "data": { 04:16:26 DEBUG "resultType": "vector", 04:16:26 DEBUG "result": [ 04:16:26 DEBUG { 04:16:26 DEBUG "metric": { 04:16:26 DEBUG "pod": "ds-idrepo-1" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG [...] 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG 04:16:26 DEBUG "pod": "ds-idrepo-0" 04:16:26 DEBUG }, 04:16:26 DEBUG "value": [ 04:16:26 DEBUG 1713064586, 04:16:26 DEBUG "0" 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG ] 04:16:26 DEBUG } 04:16:26 DEBUG } 04:16:26 INFO 04:16:26 INFO [loop_until]: kubectl --namespace=xlou top pods 04:16:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:16:26 INFO [loop_until]: OK (rc = 0) 04:16:26 DEBUG --- stdout --- 04:16:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 1247m 8440Mi am-7d597cf7d6-8vr8f 1193m 8083Mi am-7d597cf7d6-9wz9h 1469m 8813Mi ds-cts-0 2989m 7609Mi ds-cts-1 3142m 7926Mi ds-cts-2 3025m 7839Mi ds-idrepo-0 753m 4584Mi ds-idrepo-1 107m 4301Mi ds-idrepo-2 102m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 11m 996Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 9m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 4740m 8191Mi 04:16:26 DEBUG --- stderr --- 04:16:26 DEBUG 04:16:27 INFO 04:16:27 INFO [loop_until]: kubectl --namespace=xlou top node 04:16:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:16:27 INFO [loop_until]: OK (rc = 0) 04:16:27 DEBUG --- stdout --- 04:16:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4819Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3236m 10% 10024Mi 8% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1230m 4% 10488Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 155m 0% 3069Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 151m 0% 6372Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 3273m 10% 11034Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2916Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1308m 4% 9532Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 163m 0% 5527Mi 4% gke-xlou-cdm-ds-0f601751-j488 3201m 10% 9449Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2201Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 1688m 5% 10346Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 800m 2% 6075Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 662m 4% 3945Mi 6% 04:16:27 DEBUG --- stderr --- 04:16:27 DEBUG 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713064646" 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 DEBUG --- http response --- 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064646" 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 04:17:26 ERROR [http_cmd]: ERROR 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064646" 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG --- status code --- 04:17:26 ERROR --- status code --- 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713064646" 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 ERROR http status code is 400 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG --- http response --- 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG --- status code --- 04:17:26 ERROR --- http response --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:26 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 04:17:26 DEBUG { 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG "status": "success", 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG "data": { 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG { 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG { 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "result": [ 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG "data": { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG { 04:17:26 DEBUG "data": { 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:17:26 DEBUG { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG "resultType": "vector", 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG "result": [ 04:17:26 DEBUG "result": [ 04:17:26 DEBUG { 04:17:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:17:26 DEBUG "result": [ 04:17:26 DEBUG { 04:17:26 DEBUG "data": { 04:17:26 DEBUG { 04:17:26 DEBUG { 04:17:26 DEBUG "result": [ 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG { 04:17:26 DEBUG { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG }, 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG { 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:17:26 DEBUG "value": [ 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "data": { 04:17:26 DEBUG "result": [ 04:17:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:17:26 DEBUG "data": { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG "pod": "ds-idrepo-1" 04:17:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:17:26 DEBUG }, 04:17:26 DEBUG 04:17:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG { 04:17:26 DEBUG }, 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:17:26 DEBUG }, 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG }, 04:17:26 DEBUG "result": [ 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "value": [ 04:17:26 DEBUG "result": [ 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "value": [ 04:17:26 DEBUG { 04:17:26 DEBUG "instance": "10.142.0.107:9100" 04:17:26 DEBUG 04:17:26 DEBUG { 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG { 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG "metric": { 04:17:26 DEBUG }, 04:17:26 DEBUG 04:17:26 DEBUG "metric": { 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "pod": "ds-idrepo-1" 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG "instance": "10.142.0.107:9100" 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "data": { 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG }, 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG }, 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "value": [ 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG "result": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "pod": "am-7d597cf7d6-8vr8f" 04:17:26 DEBUG 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG { 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG }, 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "metric": { 04:17:26 DEBUG 04:17:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "pod": "ds-idrepo-0" 04:17:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:17:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:17:26 DEBUG }, 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG }, 04:17:26 DEBUG [...] 04:17:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:17:26 DEBUG }, 04:17:26 DEBUG }, 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG "0" 04:17:26 DEBUG }, 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG "value": [ 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG ] 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG "instance": "10.142.0.43:9100" 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "0.015" 04:17:26 DEBUG } 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG 04:17:26 DEBUG }, 04:17:26 DEBUG "0" 04:17:26 DEBUG 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "0" 04:17:26 DEBUG 04:17:26 DEBUG "70404" 04:17:26 DEBUG ] 04:17:26 DEBUG ] 04:17:26 DEBUG "1322.7333333333333" 04:17:26 DEBUG "pod": "ds-idrepo-0" 04:17:26 DEBUG "value": [ 04:17:26 DEBUG ] 04:17:26 DEBUG "instance": "10.142.0.43:9100" 04:17:26 DEBUG "0.03333333333333333" 04:17:26 DEBUG ] 04:17:26 DEBUG 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG }, 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG } 04:17:26 DEBUG }, 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG [...] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 DEBUG "value": [ 04:17:26 DEBUG "7114.433333333333" 04:17:26 DEBUG ] 04:17:26 DEBUG "value": [ 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 DEBUG "0" 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 DEBUG "12695577600" 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 DEBUG 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG ] 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 DEBUG }, 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG "value": [ 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG "115351552" 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG ] 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "pod": "ds-idrepo-1" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "pod": "ds-idrepo-0" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "0" 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "0" 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064646" 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "instance": "10.142.0.107:9100" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "instance": "10.142.0.43:9100" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "0.00030000000000001136" 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "pod": "ds-idrepo-1" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "pod": "ds-idrepo-0" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "0" 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064646" 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "pod": "am-7d597cf7d6-8ctft" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "pod": "am-7d597cf7d6-9wz9h" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "0.03333333333333333" 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "0" 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "instance": "10.142.0.107:9100" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "instance": "10.142.0.43:9100" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "218588.23333333334" 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "0" 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713064646" 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "instance": "10.142.0.107:9100" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "instance": "10.142.0.43:9100" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "0" 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713064646" 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:17:26 DEBUG { 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG "metric": { 04:17:26 DEBUG { 04:17:26 DEBUG "pod": "ds-idrepo-1" 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG }, 04:17:26 DEBUG "data": { 04:17:26 DEBUG "value": [ 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG 04:17:26 DEBUG "result": [ 04:17:26 DEBUG 04:17:26 DEBUG { 04:17:26 DEBUG 04:17:26 DEBUG "metric": { 04:17:26 DEBUG [...] 04:17:26 DEBUG "pod": "ds-idrepo-1" 04:17:26 DEBUG 04:17:26 DEBUG }, 04:17:26 DEBUG 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "pod": "ds-cts-1" 04:17:26 DEBUG 04:17:26 DEBUG }, 04:17:26 DEBUG 04:17:26 DEBUG "value": [ 04:17:26 DEBUG [...] 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG 04:17:26 DEBUG "0.0001" 04:17:26 DEBUG 04:17:26 DEBUG ] 04:17:26 DEBUG 04:17:26 DEBUG } 04:17:26 DEBUG "pod": "ds-cts-1" 04:17:26 DEBUG ] 04:17:26 DEBUG }, 04:17:26 DEBUG } 04:17:26 DEBUG "value": [ 04:17:26 DEBUG } 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "0" 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "0.028050839376284192" 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "pod": "am-7d597cf7d6-8vr8f" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "0" 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713064646" 04:17:26 INFO [http_cmd]: http status code OK 04:17:26 DEBUG --- status code --- 04:17:26 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 04:17:26 DEBUG --- http response --- 04:17:26 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 04:17:26 DEBUG ----- output ----- 04:17:26 DEBUG { 04:17:26 DEBUG "status": "success", 04:17:26 DEBUG "data": { 04:17:26 DEBUG "resultType": "vector", 04:17:26 DEBUG "result": [ 04:17:26 DEBUG { 04:17:26 DEBUG "metric": { 04:17:26 DEBUG "pod": "ds-idrepo-1" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG [...] 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG 04:17:26 DEBUG "pod": "ds-idrepo-0" 04:17:26 DEBUG }, 04:17:26 DEBUG "value": [ 04:17:26 DEBUG 1713064646, 04:17:26 DEBUG "0" 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG ] 04:17:26 DEBUG } 04:17:26 DEBUG } 04:17:26 INFO 04:17:26 INFO [loop_until]: kubectl --namespace=xlou top pods 04:17:26 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:17:26 INFO [loop_until]: OK (rc = 0) 04:17:26 DEBUG --- stdout --- 04:17:26 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 1235m 8387Mi am-7d597cf7d6-8vr8f 1198m 8064Mi am-7d597cf7d6-9wz9h 1390m 8732Mi ds-cts-0 2741m 7573Mi ds-cts-1 2923m 7916Mi ds-cts-2 2973m 7828Mi ds-idrepo-0 656m 4584Mi ds-idrepo-1 115m 4300Mi ds-idrepo-2 161m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 996Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-84dfb5c75c-qnqlj 9m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 2m 15Mi 04:17:26 DEBUG --- stderr --- 04:17:26 DEBUG 04:17:27 INFO 04:17:27 INFO [loop_until]: kubectl --namespace=xlou top node 04:17:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 04:17:27 INFO [loop_until]: OK (rc = 0) 04:17:27 DEBUG --- stdout --- 04:17:27 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4837Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3244m 10% 10026Mi 8% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1249m 4% 10491Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 93m 0% 3078Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 522m 1% 6334Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 3074m 10% 11041Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2928Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1339m 4% 9528Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 524m 1% 5467Mi 4% gke-xlou-cdm-ds-0f601751-j488 3115m 10% 9459Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 69m 0% 2212Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 1662m 5% 10328Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 1132m 3% 6074Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 1233m 7% 4412Mi 7% 04:17:27 DEBUG --- stderr --- 04:17:27 DEBUG 04:18:24 INFO Finished: True 04:18:24 INFO Waiting for threads to register finish flag 04:18:27 INFO Done. Have a nice day! :) 127.0.0.1 - - [14/Apr/2024 04:18:27] "GET /monitoring/stop HTTP/1.1" 200 - 04:18:30 WARNING File /tmp/lodemon_data-20240414_035725/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6511', ...) 127.0.0.1 - - [14/Apr/2024 04:18:34] "GET /monitoring/process HTTP/1.1" 200 -