==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-7c47b75958-dgfm9 Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Mon, 15 Apr 2024 17:43:19 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=7c47b75958 skaffold.dev/run-id=d346b44b-3161-44cf-85de-9bcac255aa7a Annotations: Status: Running IP: 10.106.46.142 IPs: IP: 10.106.46.142 Controlled By: ReplicaSet/lodemon-7c47b75958 Containers: lodemon: Container ID: containerd://fc8a1a4f907b6441a0b10a2aba6dc5914e27784eb0dbd09f370813501f2abc3e Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:06c53901427b11990b989f0a16f422c2c89c055d3393605a0d2d532652d23afc Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Mon, 15 Apr 2024 17:43:20 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-x5v8z (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-x5v8z: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 47m default-scheduler Successfully assigned xlou/lodemon-7c47b75958-dgfm9 to gke-xlou-cdm-ds-0f601751-mc5d Normal Pulling 47m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 47m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 287.057943ms (287.065552ms including waiting) Normal Created 47m kubelet Created container lodemon Normal Started 47m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.46.142:8080 Press CTRL+C to quit 18:44:27 INFO 18:44:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:27 INFO [loop_until]: OK (rc = 0) 18:44:27 DEBUG --- stdout --- 18:44:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:27 DEBUG --- stderr --- 18:44:27 DEBUG 18:44:27 INFO 18:44:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:27 INFO [loop_until]: OK (rc = 0) 18:44:27 DEBUG --- stdout --- 18:44:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:27 DEBUG --- stderr --- 18:44:27 DEBUG 18:44:27 INFO 18:44:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:27 INFO [loop_until]: OK (rc = 0) 18:44:27 DEBUG --- stdout --- 18:44:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:27 DEBUG --- stderr --- 18:44:27 DEBUG 18:44:27 INFO 18:44:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:27 INFO [loop_until]: OK (rc = 0) 18:44:27 DEBUG --- stdout --- 18:44:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:27 DEBUG --- stderr --- 18:44:27 DEBUG 18:44:27 INFO 18:44:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:27 INFO [loop_until]: OK (rc = 0) 18:44:27 DEBUG --- stdout --- 18:44:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:27 DEBUG --- stderr --- 18:44:27 DEBUG 18:44:27 INFO 18:44:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:27 INFO [loop_until]: OK (rc = 0) 18:44:27 DEBUG --- stdout --- 18:44:27 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:27 DEBUG --- stderr --- 18:44:27 DEBUG 18:44:27 INFO 18:44:27 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:27 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:28 INFO [loop_until]: OK (rc = 0) 18:44:28 DEBUG --- stdout --- 18:44:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:28 DEBUG --- stderr --- 18:44:28 DEBUG 18:44:28 INFO 18:44:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:28 INFO [loop_until]: OK (rc = 0) 18:44:28 DEBUG --- stdout --- 18:44:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:28 DEBUG --- stderr --- 18:44:28 DEBUG 18:44:28 INFO 18:44:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:28 INFO [loop_until]: OK (rc = 0) 18:44:28 DEBUG --- stdout --- 18:44:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:28 DEBUG --- stderr --- 18:44:28 DEBUG 18:44:28 INFO 18:44:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:28 INFO [loop_until]: OK (rc = 0) 18:44:28 DEBUG --- stdout --- 18:44:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:28 DEBUG --- stderr --- 18:44:28 DEBUG 18:44:28 INFO 18:44:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:28 INFO [loop_until]: OK (rc = 0) 18:44:28 DEBUG --- stdout --- 18:44:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:28 DEBUG --- stderr --- 18:44:28 DEBUG 18:44:28 INFO 18:44:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:28 INFO [loop_until]: OK (rc = 0) 18:44:28 DEBUG --- stdout --- 18:44:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:28 DEBUG --- stderr --- 18:44:28 DEBUG 18:44:28 INFO 18:44:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:28 INFO [loop_until]: OK (rc = 0) 18:44:28 DEBUG --- stdout --- 18:44:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:28 DEBUG --- stderr --- 18:44:28 DEBUG 18:44:28 INFO 18:44:28 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:28 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:28 INFO [loop_until]: OK (rc = 0) 18:44:28 DEBUG --- stdout --- 18:44:28 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:28 DEBUG --- stderr --- 18:44:28 DEBUG 18:44:29 INFO 18:44:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:29 INFO [loop_until]: OK (rc = 0) 18:44:29 DEBUG --- stdout --- 18:44:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:29 DEBUG --- stderr --- 18:44:29 DEBUG 18:44:29 INFO 18:44:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:29 INFO [loop_until]: OK (rc = 0) 18:44:29 DEBUG --- stdout --- 18:44:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:29 DEBUG --- stderr --- 18:44:29 DEBUG 18:44:29 INFO 18:44:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:29 INFO [loop_until]: OK (rc = 0) 18:44:29 DEBUG --- stdout --- 18:44:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:29 DEBUG --- stderr --- 18:44:29 DEBUG 18:44:29 INFO 18:44:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:29 INFO [loop_until]: OK (rc = 0) 18:44:29 DEBUG --- stdout --- 18:44:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:29 DEBUG --- stderr --- 18:44:29 DEBUG 18:44:29 INFO 18:44:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:29 INFO [loop_until]: OK (rc = 0) 18:44:29 DEBUG --- stdout --- 18:44:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:29 DEBUG --- stderr --- 18:44:29 DEBUG 18:44:29 INFO 18:44:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:29 INFO [loop_until]: OK (rc = 0) 18:44:29 DEBUG --- stdout --- 18:44:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:29 DEBUG --- stderr --- 18:44:29 DEBUG 18:44:29 INFO 18:44:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:29 INFO [loop_until]: OK (rc = 0) 18:44:29 DEBUG --- stdout --- 18:44:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:29 DEBUG --- stderr --- 18:44:29 DEBUG 18:44:30 INFO 18:44:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:30 INFO [loop_until]: OK (rc = 0) 18:44:30 DEBUG --- stdout --- 18:44:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:30 DEBUG --- stderr --- 18:44:30 DEBUG 18:44:30 INFO 18:44:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:30 INFO [loop_until]: OK (rc = 0) 18:44:30 DEBUG --- stdout --- 18:44:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:30 DEBUG --- stderr --- 18:44:30 DEBUG 18:44:30 INFO 18:44:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:30 INFO [loop_until]: OK (rc = 0) 18:44:30 DEBUG --- stdout --- 18:44:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:30 DEBUG --- stderr --- 18:44:30 DEBUG 18:44:30 INFO 18:44:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:30 INFO [loop_until]: OK (rc = 0) 18:44:30 DEBUG --- stdout --- 18:44:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:30 DEBUG --- stderr --- 18:44:30 DEBUG 18:44:30 INFO 18:44:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:30 INFO [loop_until]: OK (rc = 0) 18:44:30 DEBUG --- stdout --- 18:44:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:30 DEBUG --- stderr --- 18:44:30 DEBUG 18:44:30 INFO 18:44:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 18:44:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:30 INFO [loop_until]: OK (rc = 0) 18:44:30 DEBUG --- stdout --- 18:44:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 18:44:30 DEBUG --- stderr --- 18:44:30 DEBUG 18:44:30 INFO Initializing monitoring instance threads 18:44:30 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 18:44:30 INFO Starting instance threads 18:44:30 INFO 18:44:30 INFO Thread started 18:44:30 INFO [loop_until]: kubectl --namespace=xlou top node 18:44:30 INFO 18:44:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:30 INFO Thread started 18:44:30 INFO [loop_until]: kubectl --namespace=xlou top pods 18:44:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO Thread started 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG --- status code --- 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 INFO Thread started 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG --- http response --- 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 DEBUG --- http response --- 18:44:30 INFO Thread started 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 DEBUG ----- output ----- 18:44:30 INFO Thread started 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG { 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 DEBUG ----- output ----- 18:44:30 INFO Thread started 18:44:30 DEBUG { 18:44:30 DEBUG { 18:44:30 DEBUG "status": "success", 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 DEBUG { 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO Thread started 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG "data": { 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG --- status code --- 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 DEBUG --- status code --- 18:44:30 INFO [loop_until]: OK (rc = 0) 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 DEBUG --- status code --- 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG "data": { 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG "data": { 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG --- stdout --- 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG "result": [ 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8322Mi am-7d597cf7d6-8vr8f 7m 8434Mi am-7d597cf7d6-9wz9h 6m 9087Mi ds-cts-0 14m 8123Mi ds-cts-1 12m 8265Mi ds-cts-2 13m 8432Mi ds-idrepo-0 11m 4765Mi ds-idrepo-1 10m 4340Mi ds-idrepo-2 11m 4112Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-7c47b75958-dgfm9 2m 54Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 662Mi 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:44:30 DEBUG "data": { 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG --- http response --- 18:44:30 INFO Thread started 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:44:30 DEBUG "result": [ 18:44:30 DEBUG { 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:44:30 DEBUG "result": [ 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:44:30 DEBUG --- http response --- 18:44:30 INFO [loop_until]: OK (rc = 0) 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:44:30 DEBUG --- stderr --- 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:44:30 DEBUG { 18:44:30 DEBUG "metric": { 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG { 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG --- stdout --- 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:44:30 DEBUG ----- output ----- 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 DEBUG 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG { 18:44:30 DEBUG "result": [ 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG ----- output ----- 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203070" 18:44:30 DEBUG { 18:44:30 INFO Thread started 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG "metric": { 18:44:30 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:44:30 DEBUG { 18:44:30 DEBUG "metric": { 18:44:30 DEBUG { 18:44:30 DEBUG { 18:44:30 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5123Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 70m 0% 10402Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 57m 0% 10378Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3164Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 64m 0% 7103Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 158m 0% 12528Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2985Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 62m 0% 9963Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 67m 0% 6251Mi 5% gke-xlou-cdm-ds-0f601751-j488 66m 0% 9928Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 68m 0% 2461Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 87m 0% 10720Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 66m 0% 6894Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 67m 0% 4548Mi 7% 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG { 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG { 18:44:30 DEBUG { 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG { 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG { 18:44:30 DEBUG { 18:44:30 DEBUG { 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG { 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203070" 18:44:30 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:44:30 INFO Thread started 18:44:30 DEBUG }, 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG --- stderr --- 18:44:30 DEBUG { 18:44:30 DEBUG { 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG "data": { 18:44:30 DEBUG "metric": { 18:44:30 DEBUG { 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG "data": { 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG }, 18:44:30 DEBUG "value": [ 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203070" 18:44:30 DEBUG "data": { 18:44:30 INFO Thread started 18:44:30 DEBUG }, 18:44:30 DEBUG "data": { 18:44:30 DEBUG "data": { 18:44:30 DEBUG 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG "data": { 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG "data": { 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 DEBUG "data": { 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG "data": { 18:44:30 DEBUG "data": { 18:44:30 DEBUG "data": { 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG "data": { 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 18:44:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203070" 18:44:30 INFO Thread started 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG "data": { 18:44:30 DEBUG "data": { 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG "resultType": "vector", 18:44:30 ERROR [http_cmd]: ERROR 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG "result": [ 18:44:30 DEBUG }, 18:44:30 DEBUG "data": { 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG "result": [ 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG "result": [ 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG 18:44:30 INFO All threads has been started 127.0.0.1 - - [15/Apr/2024 18:44:30] "GET /monitoring/start HTTP/1.1" 200 - 18:44:30 DEBUG "result": [ 18:44:30 DEBUG "resultType": "vector", 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG "result": [ 18:44:30 DEBUG "result": [ 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:44:30 ERROR --- status code --- 18:44:30 DEBUG { 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG "result": [ 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:44:30 DEBUG { 18:44:30 DEBUG "value": [ 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG "result": [ 18:44:30 DEBUG "result": [ 18:44:30 DEBUG "result": [ 18:44:30 DEBUG { 18:44:30 DEBUG "result": [ 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG { 18:44:30 DEBUG "result": [ 18:44:30 DEBUG 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 DEBUG { 18:44:30 DEBUG "result": [ 18:44:30 DEBUG { 18:44:30 DEBUG "metric": { 18:44:30 DEBUG { 18:44:30 DEBUG { 18:44:30 DEBUG ----- output ----- 18:44:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG { 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG "metric": { 18:44:30 DEBUG 18:44:30 DEBUG "result": [ 18:44:30 DEBUG { 18:44:30 DEBUG { 18:44:30 DEBUG { 18:44:30 DEBUG "metric": { 18:44:30 DEBUG { 18:44:30 DEBUG 18:44:30 DEBUG [...] 18:44:30 DEBUG "metric": { 18:44:30 DEBUG { 18:44:30 DEBUG [...] 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG "metric": { 18:44:30 DEBUG "result": [ 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:44:30 DEBUG "metric": { 18:44:30 DEBUG "metric": { 18:44:30 DEBUG { 18:44:30 ERROR --- http response --- 18:44:30 DEBUG "data": { 18:44:30 DEBUG "metric": { 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:44:30 DEBUG { 18:44:30 DEBUG "pod": "ds-idrepo-1" 18:44:30 DEBUG 18:44:30 DEBUG { 18:44:30 DEBUG "metric": { 18:44:30 DEBUG "metric": { 18:44:30 DEBUG "metric": { 18:44:30 DEBUG "instance": "10.142.0.107:9100" 18:44:30 DEBUG "metric": { 18:44:30 DEBUG [...] 18:44:30 DEBUG 18:44:30 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:44:30 DEBUG "metric": { 18:44:30 DEBUG 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:44:30 DEBUG { 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG }, 18:44:30 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:44:30 DEBUG "pod": "ds-idrepo-1" 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:44:30 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG }, 18:44:30 DEBUG 18:44:30 DEBUG "metric": { 18:44:30 DEBUG "pod": "ds-idrepo-1" 18:44:30 DEBUG "pod": "ds-idrepo-1" 18:44:30 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:44:30 DEBUG }, 18:44:30 DEBUG "instance": "10.142.0.107:9100" 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG }, 18:44:30 DEBUG "pod": "ds-idrepo-1" 18:44:30 DEBUG 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG }, 18:44:30 DEBUG "metric": { 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG "value": [ 18:44:30 DEBUG }, 18:44:30 DEBUG }, 18:44:30 DEBUG "data": { 18:44:30 DEBUG "result": [ 18:44:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:44:30 DEBUG }, 18:44:30 DEBUG { 18:44:30 DEBUG "data": { 18:44:30 DEBUG "value": [ 18:44:30 DEBUG [...] 18:44:30 DEBUG "instance": "10.142.0.107:9100" 18:44:30 INFO [http_cmd]: http status code OK 18:44:30 DEBUG }, 18:44:30 DEBUG }, 18:44:30 DEBUG }, 18:44:30 DEBUG "value": [ 18:44:30 DEBUG }, 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG "value": [ 18:44:30 DEBUG }, 18:44:30 DEBUG 18:44:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:44:30 DEBUG "value": [ 18:44:30 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:44:30 DEBUG 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:44:30 DEBUG "value": [ 18:44:30 DEBUG "value": [ 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG { 18:44:30 DEBUG "value": [ 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG }, 18:44:30 DEBUG --- status code --- 18:44:30 DEBUG "value": [ 18:44:30 DEBUG "value": [ 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 18:44:30 DEBUG "pod": "ds-cts-0" 18:44:30 DEBUG 18:44:30 DEBUG "value": [ 18:44:30 DEBUG "pod": "ds-cts-0" 18:44:30 DEBUG 18:44:30 DEBUG }, 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG "result": [ 18:44:30 DEBUG "metric": { 18:44:30 DEBUG 18:44:30 DEBUG "data": { 18:44:30 DEBUG "result": [ 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG "pod": "ds-cts-0" 18:44:30 DEBUG }, 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG }, 18:44:30 DEBUG 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG { 18:44:30 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:44:30 DEBUG 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG { 18:44:30 DEBUG 18:44:30 DEBUG "value": [ 18:44:30 DEBUG "pod": "ds-cts-0" 18:44:30 DEBUG --- http response --- 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG }, 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 18:44:30 DEBUG { 18:44:30 DEBUG 18:44:30 DEBUG [...] 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG "metric": { 18:44:30 DEBUG "pod": "ds-idrepo-1" 18:44:30 DEBUG }, 18:44:30 DEBUG "result": [ 18:44:30 DEBUG "metric": { 18:44:30 DEBUG [...] 18:44:30 DEBUG 18:44:30 DEBUG }, 18:44:30 DEBUG 18:44:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:44:30 DEBUG 18:44:30 DEBUG [...] 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG "0.008485650857733295" 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG [...] 18:44:30 DEBUG [...] 18:44:30 DEBUG [...] 18:44:30 DEBUG 18:44:30 DEBUG }, 18:44:30 DEBUG "value": [ 18:44:30 DEBUG { 18:44:30 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG "value": [ 18:44:30 DEBUG [...] 18:44:30 DEBUG ----- output ----- 18:44:30 DEBUG 18:44:30 DEBUG "pod": "ds-idrepo-0" 18:44:30 DEBUG 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG [...] 18:44:30 DEBUG "value": [ 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG 18:44:30 DEBUG [...] 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG [...] 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 18:44:30 DEBUG "metric": { 18:44:30 DEBUG }, 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG 18:44:30 DEBUG { 18:44:30 DEBUG [...] 18:44:30 DEBUG }, 18:44:30 DEBUG [...] 18:44:30 DEBUG "0" 18:44:30 DEBUG [...] 18:44:30 DEBUG [...] 18:44:30 DEBUG 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG "data": { 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG "pod": "ds-idrepo-1" 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 18:44:30 DEBUG [...] 18:44:30 DEBUG "219345819.48948926" 18:44:30 DEBUG 18:44:30 DEBUG "status": "success", 18:44:30 DEBUG "data": { 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 18:44:30 DEBUG ] 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG "843.8600919453365" 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG }, 18:44:30 DEBUG 18:44:30 DEBUG "pod": "ds-idrepo-0" 18:44:30 DEBUG 18:44:30 DEBUG ] 18:44:30 DEBUG 18:44:30 DEBUG "pod": "ds-idrepo-0" 18:44:30 DEBUG "resultType": "vector", 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG 18:44:30 DEBUG } 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG ] 18:44:30 DEBUG "result": [ 18:44:30 DEBUG "instance": "10.142.0.43:9100" 18:44:30 DEBUG 18:44:30 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:44:30 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:44:30 DEBUG "pod": "ds-idrepo-0" 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG [...] 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 18:44:30 DEBUG }, 18:44:30 DEBUG 18:44:30 DEBUG } 18:44:30 DEBUG 18:44:30 DEBUG }, 18:44:30 DEBUG "result": [ 18:44:30 DEBUG "0" 18:44:30 DEBUG 18:44:30 DEBUG ] 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:44:30 DEBUG } 18:44:30 DEBUG { 18:44:30 DEBUG }, 18:44:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:44:30 DEBUG }, 18:44:30 DEBUG }, 18:44:30 DEBUG }, 18:44:30 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:44:30 DEBUG [...] 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 18:44:30 DEBUG ] 18:44:30 DEBUG 18:44:30 DEBUG "value": [ 18:44:30 DEBUG { 18:44:30 DEBUG ] 18:44:30 DEBUG "pod": "ds-idrepo-0" 18:44:30 DEBUG } 18:44:30 DEBUG "instance": "10.142.0.43:9100" 18:44:30 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:44:30 DEBUG }, 18:44:30 DEBUG ] 18:44:30 DEBUG "metric": { 18:44:30 DEBUG "value": [ 18:44:30 DEBUG }, 18:44:30 DEBUG "value": [ 18:44:30 DEBUG "value": [ 18:44:30 DEBUG "value": [ 18:44:30 DEBUG }, 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG 18:44:30 DEBUG [...] 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG "instance": "10.142.0.43:9100" 18:44:30 DEBUG } 18:44:30 DEBUG 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG "metric": { 18:44:30 DEBUG } 18:44:30 DEBUG }, 18:44:30 DEBUG } 18:44:30 DEBUG }, 18:44:30 DEBUG }, 18:44:30 DEBUG "value": [ 18:44:30 DEBUG } 18:44:30 DEBUG "instance": "10.142.0.107:9100" 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG "791433216" 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG "0" 18:44:30 DEBUG }, 18:44:30 DEBUG } 18:44:30 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:44:30 DEBUG "0" 18:44:30 DEBUG "instance": "10.142.0.107:9100" 18:44:30 DEBUG ] 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG } 18:44:30 DEBUG }, 18:44:30 DEBUG "0.00033333333333333365" 18:44:30 DEBUG ] 18:44:30 DEBUG "0" 18:44:30 DEBUG "0" 18:44:30 DEBUG "0" 18:44:30 DEBUG ] 18:44:30 DEBUG 18:44:30 DEBUG ] 18:44:30 DEBUG [...] 18:44:30 DEBUG 18:44:30 DEBUG ] 18:44:30 DEBUG "value": [ 18:44:30 DEBUG }, 18:44:30 DEBUG ] 18:44:30 DEBUG }, 18:44:30 DEBUG } 18:44:30 DEBUG "value": [ 18:44:30 DEBUG "0" 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG "0" 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG ] 18:44:30 DEBUG 18:44:30 DEBUG } 18:44:30 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:44:30 DEBUG } 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG "value": [ 18:44:30 DEBUG } 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG ] 18:44:30 DEBUG "5164.2" 18:44:30 DEBUG ] 18:44:30 DEBUG 18:44:30 DEBUG "0.03124835270160765" 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG 18:44:30 DEBUG ] 18:44:30 DEBUG }, 18:44:30 DEBUG ] 18:44:30 DEBUG 18:44:30 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:44:30 DEBUG } 18:44:30 DEBUG "value": [ 18:44:30 DEBUG } 18:44:30 DEBUG "6834.5" 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG "pod": "ds-cts-2" 18:44:30 DEBUG } 18:44:30 DEBUG "value": [ 18:44:30 DEBUG } 18:44:30 DEBUG 18:44:30 DEBUG "pod": "ds-cts-2" 18:44:30 DEBUG }, 18:44:30 DEBUG ] 18:44:30 DEBUG ] 18:44:30 DEBUG 18:44:30 DEBUG "0" 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG }, 18:44:30 DEBUG } 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG } 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG }, 18:44:30 DEBUG "value": [ 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG [...] 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG [...] 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG "value": [ 18:44:30 DEBUG "2" 18:44:30 DEBUG "0" 18:44:30 DEBUG "value": [ 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG 18:44:30 DEBUG 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG "0" 18:44:30 DEBUG ] 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG "0.076" 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG 18:44:30 DEBUG } 18:44:30 DEBUG 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG "0" 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG "instance": "10.142.0.43:9100" 18:44:30 DEBUG 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG }, 18:44:30 DEBUG "instance": "10.142.0.43:9100" 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG "value": [ 18:44:30 DEBUG }, 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG "value": [ 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG "0.00036666666666652263" 18:44:30 DEBUG 1713203070, 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG "13758241792" 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG ] 18:44:30 DEBUG } 18:44:30 DEBUG } 18:44:30 DEBUG } 18:45:30 INFO 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203130" 18:45:30 INFO [loop_until]: kubectl --namespace=xlou top pods 18:45:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 INFO 18:45:30 INFO [loop_until]: kubectl --namespace=xlou top node 18:45:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203130" 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 DEBUG --- status code --- 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG --- status code --- 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 DEBUG --- status code --- 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 ERROR [http_cmd]: ERROR 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG --- http response --- 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG --- http response --- 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 DEBUG --- http response --- 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 ERROR --- status code --- 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG --- status code --- 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203130" 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203130" 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203130" 18:45:30 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG --- http response --- 18:45:30 INFO [loop_until]: OK (rc = 0) 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG --- http response --- 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG { 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG --- status code --- 18:45:30 ERROR --- http response --- 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG --- stdout --- 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:45:30 DEBUG { 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:45:30 DEBUG { 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG { 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG { 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:45:30 DEBUG { 18:45:30 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 2402m 8325Mi am-7d597cf7d6-8vr8f 4258m 8443Mi am-7d597cf7d6-9wz9h 2391m 9091Mi ds-cts-0 1239m 8132Mi ds-cts-1 2047m 8282Mi ds-cts-2 708m 8443Mi ds-idrepo-0 2819m 4766Mi ds-idrepo-1 707m 4340Mi ds-idrepo-2 495m 4112Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1022Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-7c47b75958-dgfm9 2m 61Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 3404m 1090Mi 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG { 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG { 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG "data": { 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:45:30 INFO [http_cmd]: http status code OK 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG --- stderr --- 18:45:30 DEBUG { 18:45:30 DEBUG "data": { 18:45:30 DEBUG { 18:45:30 DEBUG { 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG { 18:45:30 DEBUG "data": { 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG "data": { 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG --- status code --- 18:45:30 DEBUG "data": { 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:45:30 DEBUG "data": { 18:45:30 DEBUG { 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:45:30 DEBUG "data": { 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG "data": { 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG "result": [ 18:45:30 DEBUG { 18:45:30 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG { 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG "data": { 18:45:30 DEBUG "result": [ 18:45:30 DEBUG "data": { 18:45:30 DEBUG "data": { 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG "data": { 18:45:30 DEBUG "result": [ 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:45:30 DEBUG { 18:45:30 DEBUG "result": [ 18:45:30 DEBUG { 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG --- http response --- 18:45:30 DEBUG "result": [ 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG { 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG "result": [ 18:45:30 DEBUG "data": { 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:45:30 DEBUG { 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG { 18:45:30 DEBUG "result": [ 18:45:30 DEBUG { 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG { 18:45:30 DEBUG "result": [ 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG { 18:45:30 DEBUG "metric": { 18:45:30 DEBUG "data": { 18:45:30 DEBUG { 18:45:30 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:45:30 DEBUG "data": { 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG { 18:45:30 DEBUG { 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG "result": [ 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG "metric": { 18:45:30 DEBUG "result": [ 18:45:30 DEBUG "result": [ 18:45:30 DEBUG { 18:45:30 DEBUG { 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG { 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG "data": { 18:45:30 DEBUG "metric": { 18:45:30 DEBUG { 18:45:30 DEBUG { 18:45:30 DEBUG "data": { 18:45:30 DEBUG "metric": { 18:45:30 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG "metric": { 18:45:30 DEBUG ----- output ----- 18:45:30 DEBUG "resultType": "vector", 18:45:30 DEBUG "data": { 18:45:30 DEBUG "status": "success", 18:45:30 DEBUG "metric": { 18:45:31 DEBUG "result": [ 18:45:31 DEBUG { 18:45:31 DEBUG { 18:45:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:45:31 DEBUG { 18:45:31 DEBUG { 18:45:31 DEBUG "status": "success", 18:45:31 DEBUG "status": "success", 18:45:31 DEBUG "data": { 18:45:31 DEBUG "metric": { 18:45:31 DEBUG "result": [ 18:45:31 DEBUG "resultType": "vector", 18:45:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:45:31 DEBUG "metric": { 18:45:31 DEBUG "status": "success", 18:45:31 DEBUG "resultType": "vector", 18:45:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:45:31 DEBUG }, 18:45:31 DEBUG "result": [ 18:45:31 DEBUG "pod": "ds-idrepo-1" 18:45:31 DEBUG { 18:45:31 DEBUG "result": [ 18:45:31 DEBUG "resultType": "vector", 18:45:31 DEBUG "data": { 18:45:31 DEBUG "pod": "ds-idrepo-1" 18:45:31 DEBUG { 18:45:31 DEBUG "metric": { 18:45:31 DEBUG "status": "success", 18:45:31 DEBUG }, 18:45:31 DEBUG "metric": { 18:45:31 DEBUG "metric": { 18:45:31 DEBUG "data": { 18:45:31 DEBUG "data": { 18:45:31 DEBUG "resultType": "vector", 18:45:31 DEBUG "pod": "ds-idrepo-1" 18:45:31 DEBUG { 18:45:31 DEBUG "result": [ 18:45:31 DEBUG }, 18:45:31 DEBUG "pod": "ds-idrepo-1" 18:45:31 DEBUG "data": { 18:45:31 DEBUG "result": [ 18:45:31 DEBUG }, 18:45:31 DEBUG "value": [ 18:45:31 DEBUG { 18:45:31 DEBUG }, 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "status": "success", 18:45:31 DEBUG "result": [ 18:45:31 DEBUG "resultType": "vector", 18:45:31 DEBUG }, 18:45:31 DEBUG "metric": { 18:45:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:45:31 DEBUG "data": { 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "instance": "10.142.0.107:9100" 18:45:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:45:31 DEBUG "resultType": "vector", 18:45:31 DEBUG "resultType": "vector", 18:45:31 DEBUG "result": [ 18:45:31 DEBUG }, 18:45:31 DEBUG "metric": { 18:45:31 DEBUG { 18:45:31 DEBUG "value": [ 18:45:31 DEBUG }, 18:45:31 DEBUG "resultType": "vector", 18:45:31 DEBUG { 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 18:45:31 DEBUG "metric": { 18:45:31 DEBUG { 18:45:31 DEBUG 18:45:31 DEBUG "data": { 18:45:31 DEBUG { 18:45:31 DEBUG "result": [ 18:45:31 DEBUG { 18:45:31 DEBUG "instance": "10.142.0.107:9100" 18:45:31 DEBUG }, 18:45:31 DEBUG "resultType": "vector", 18:45:31 DEBUG 18:45:31 DEBUG }, 18:45:31 DEBUG }, 18:45:31 DEBUG "result": [ 18:45:31 DEBUG "result": [ 18:45:31 DEBUG { 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "instance": "10.142.0.107:9100" 18:45:31 DEBUG "metric": { 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "result": [ 18:45:31 DEBUG "metric": { 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:45:31 DEBUG "metric": { 18:45:31 DEBUG 18:45:31 DEBUG "resultType": "vector", 18:45:31 DEBUG "metric": { 18:45:31 DEBUG "metric": { 18:45:31 DEBUG "value": [ 18:45:31 DEBUG }, 18:45:31 INFO [loop_until]: OK (rc = 0) 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "result": [ 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "value": [ 18:45:31 DEBUG { 18:45:31 DEBUG { 18:45:31 DEBUG "metric": { 18:45:31 DEBUG 18:45:31 DEBUG }, 18:45:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG { 18:45:31 DEBUG "pod": "ds-idrepo-1" 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG }, 18:45:31 DEBUG "instance": "10.142.0.107:9100" 18:45:31 DEBUG 18:45:31 DEBUG "result": [ 18:45:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:45:31 DEBUG "pod": "ds-idrepo-1" 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG --- stdout --- 18:45:31 DEBUG 18:45:31 DEBUG { 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "metric": { 18:45:31 DEBUG "metric": { 18:45:31 DEBUG "instance": "10.142.0.107:9100" 18:45:31 DEBUG }, 18:45:31 DEBUG "value": [ 18:45:31 DEBUG }, 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "metric": { 18:45:31 DEBUG }, 18:45:31 DEBUG 18:45:31 DEBUG [...] 18:45:31 DEBUG "value": [ 18:45:31 DEBUG }, 18:45:31 DEBUG [...] 18:45:31 DEBUG { 18:45:31 DEBUG }, 18:45:31 DEBUG }, 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5127Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 1584m 5% 10430Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 4111m 13% 10370Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 4341m 14% 3103Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 879m 2% 7147Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 1968m 6% 12510Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2984Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 3219m 10% 9970Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 632m 2% 6276Mi 5% gke-xlou-cdm-ds-0f601751-j488 2306m 7% 9977Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 67m 0% 2449Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 5652m 18% 10729Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 6008m 20% 6915Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 3323m 20% 4969Mi 8% 18:45:31 DEBUG 18:45:31 DEBUG "metric": { 18:45:31 DEBUG [...] 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:45:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG [...] 18:45:31 DEBUG 18:45:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:45:31 DEBUG "value": [ 18:45:31 DEBUG [...] 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 18:45:31 DEBUG "metric": { 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG --- stderr --- 18:45:31 DEBUG 18:45:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG }, 18:45:31 DEBUG }, 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG [...] 18:45:31 DEBUG }, 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "pod": "ds-idrepo-1" 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG [...] 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG [...] 18:45:31 DEBUG }, 18:45:31 DEBUG 18:45:31 DEBUG [...] 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "value": [ 18:45:31 DEBUG [...] 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG }, 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG [...] 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 18:45:31 DEBUG [...] 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG [...] 18:45:31 DEBUG [...] 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "pod": "am-7d597cf7d6-8vr8f" 18:45:31 DEBUG [...] 18:45:31 DEBUG 18:45:31 DEBUG "pod": "ds-idrepo-0" 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG [...] 18:45:31 DEBUG "pod": "am-7d597cf7d6-8vr8f" 18:45:31 DEBUG }, 18:45:31 DEBUG 18:45:31 DEBUG [...] 18:45:31 DEBUG }, 18:45:31 DEBUG 18:45:31 DEBUG [...] 18:45:31 DEBUG [...] 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG }, 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG [...] 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG }, 18:45:31 DEBUG "pod": "ds-idrepo-0" 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG }, 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "pod": "ds-idrepo-0" 18:45:31 DEBUG 18:45:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "instance": "10.142.0.43:9100" 18:45:31 DEBUG "pod": "am-7d597cf7d6-8vr8f" 18:45:31 DEBUG [...] 18:45:31 DEBUG [...] 18:45:31 DEBUG "pod": "ds-cts-2" 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG }, 18:45:31 DEBUG [...] 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG }, 18:45:31 DEBUG "instance": "10.142.0.43:9100" 18:45:31 DEBUG }, 18:45:31 DEBUG [...] 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG }, 18:45:31 DEBUG }, 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG }, 18:45:31 DEBUG 18:45:31 DEBUG "instance": "10.142.0.43:9100" 18:45:31 DEBUG "instance": "10.142.0.43:9100" 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG "0.1339031207802454" 18:45:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:45:31 DEBUG 18:45:31 DEBUG "0" 18:45:31 DEBUG [...] 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG }, 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 18:45:31 DEBUG }, 18:45:31 DEBUG }, 18:45:31 DEBUG "737.0666666666666" 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG 18:45:31 DEBUG "pod": "ds-idrepo-0" 18:45:31 DEBUG "567.4898687264036" 18:45:31 DEBUG ] 18:45:31 DEBUG }, 18:45:31 DEBUG "instance": "10.142.0.43:9100" 18:45:31 DEBUG ] 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG "pod": "ds-idrepo-0" 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG "value": [ 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG 18:45:31 DEBUG "1091969024" 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG 18:45:31 DEBUG 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "value": [ 18:45:31 DEBUG ] 18:45:31 DEBUG "0" 18:45:31 DEBUG 18:45:31 DEBUG }, 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG "value": [ 18:45:31 DEBUG }, 18:45:31 DEBUG } 18:45:31 DEBUG "pod": "am-7d597cf7d6-8vr8f" 18:45:31 DEBUG 18:45:31 DEBUG }, 18:45:31 DEBUG "0" 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG "736.3333333333334" 18:45:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:45:31 DEBUG ] 18:45:31 DEBUG "5305.599999999999" 18:45:31 DEBUG "79022.96422015171" 18:45:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:45:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:45:31 DEBUG "0" 18:45:31 DEBUG }, 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:45:31 DEBUG "value": [ 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG "value": [ 18:45:31 DEBUG ] 18:45:31 DEBUG }, 18:45:31 DEBUG "pod": "ds-cts-2" 18:45:31 DEBUG "value": [ 18:45:31 DEBUG ] 18:45:31 DEBUG "0.0009333333333335266" 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG ] 18:45:31 DEBUG }, 18:45:31 DEBUG }, 18:45:31 DEBUG ] 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "5217.333333333334" 18:45:31 DEBUG "0.00033333333333333365" 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG }, 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG "0.076" 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG } 18:45:31 DEBUG "value": [ 18:45:31 DEBUG }, 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG }, 18:45:31 DEBUG ] 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "value": [ 18:45:31 DEBUG } 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG ] 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "0" 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG "13760664576" 18:45:31 DEBUG } 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG "value": [ 18:45:31 DEBUG "0" 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG "value": [ 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG ] 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG ] 18:45:31 DEBUG "737.1333333333333" 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG "0" 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG 1713203130, 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG "2.3802407840404514" 18:45:31 DEBUG "0" 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG ] 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG "0" 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG "0.00001811333333333333" 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG "11033.5" 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG ] 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:45:31 DEBUG } 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO 18:46:31 INFO [loop_until]: kubectl --namespace=xlou top pods 18:46:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203191" 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 ERROR [http_cmd]: ERROR 18:46:31 INFO 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 DEBUG --- status code --- 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 ERROR --- status code --- 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [loop_until]: kubectl --namespace=xlou top node 18:46:31 DEBUG --- status code --- 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203191" 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG --- http response --- 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 DEBUG --- status code --- 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 DEBUG --- status code --- 18:46:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203191" 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG --- status code --- 18:46:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:46:31 DEBUG --- status code --- 18:46:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 INFO [loop_until]: OK (rc = 0) 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG --- http response --- 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 ERROR --- http response --- 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG --- http response --- 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG --- stdout --- 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG --- status code --- 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG --- http response --- 18:46:31 INFO [http_cmd]: http status code OK 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:46:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 9912m 8331Mi am-7d597cf7d6-8vr8f 9021m 8446Mi am-7d597cf7d6-9wz9h 12097m 8648Mi ds-cts-0 6315m 8176Mi ds-cts-1 4537m 8341Mi ds-cts-2 4358m 8479Mi ds-idrepo-0 11151m 4756Mi ds-idrepo-1 1814m 4341Mi ds-idrepo-2 1697m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1023Mi idm-b85f9d978-rl76f 4m 1223Mi lodemon-7c47b75958-dgfm9 2m 61Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6699m 1118Mi 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:46:31 DEBUG --- status code --- 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:46:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG --- stderr --- 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:46:31 DEBUG { 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG { 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG { 18:46:31 DEBUG { 18:46:31 DEBUG { 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:46:31 DEBUG { 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:46:31 DEBUG { 18:46:31 DEBUG 18:46:31 DEBUG { 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:46:31 DEBUG { 18:46:31 DEBUG { 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG { 18:46:31 DEBUG { 18:46:31 DEBUG --- http response --- 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:46:31 DEBUG { 18:46:31 DEBUG { 18:46:31 DEBUG { 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:46:31 DEBUG { 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "data": { 18:46:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:46:31 DEBUG "data": { 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "data": { 18:46:31 DEBUG "data": { 18:46:31 DEBUG "data": { 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG { 18:46:31 DEBUG "data": { 18:46:31 DEBUG { 18:46:31 DEBUG "data": { 18:46:31 DEBUG "data": { 18:46:31 DEBUG { 18:46:31 DEBUG "data": { 18:46:31 DEBUG "data": { 18:46:31 DEBUG { 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG "data": { 18:46:31 DEBUG "data": { 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG ----- output ----- 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG { 18:46:31 DEBUG "data": { 18:46:31 DEBUG "data": { 18:46:31 DEBUG "data": { 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG { 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG { 18:46:31 DEBUG "data": { 18:46:31 DEBUG { 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "result": [ 18:46:31 DEBUG { 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "data": { 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "data": { 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "data": { 18:46:31 DEBUG "result": [ 18:46:31 DEBUG { 18:46:31 DEBUG "data": { 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "result": [ 18:46:31 DEBUG { 18:46:31 DEBUG "status": "success", 18:46:31 DEBUG { 18:46:31 DEBUG "data": { 18:46:31 DEBUG "result": [ 18:46:31 DEBUG { 18:46:31 DEBUG "result": [ 18:46:31 DEBUG { 18:46:31 DEBUG { 18:46:31 DEBUG { 18:46:31 DEBUG "data": { 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG { 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG { 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "instance": "10.142.0.107:9100" 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "data": { 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "data": { 18:46:31 DEBUG { 18:46:31 DEBUG { 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "data": { 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "result": [ 18:46:31 DEBUG { 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "result": [ 18:46:31 DEBUG { 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG }, 18:46:31 DEBUG { 18:46:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG { 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:46:31 DEBUG "resultType": "vector", 18:46:31 DEBUG "instance": "10.142.0.107:9100" 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:46:31 DEBUG { 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:46:31 DEBUG "pod": "ds-idrepo-1" 18:46:31 DEBUG "result": [ 18:46:31 DEBUG { 18:46:31 DEBUG "instance": "10.142.0.107:9100" 18:46:31 DEBUG { 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "metric": { 18:46:31 DEBUG }, 18:46:31 DEBUG { 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "result": [ 18:46:31 DEBUG "pod": "ds-idrepo-1" 18:46:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:46:31 DEBUG }, 18:46:31 DEBUG "result": [ 18:46:31 DEBUG }, 18:46:31 DEBUG { 18:46:31 DEBUG }, 18:46:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:46:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:46:31 DEBUG }, 18:46:31 DEBUG "value": [ 18:46:31 DEBUG }, 18:46:31 DEBUG { 18:46:31 DEBUG "metric": { 18:46:31 DEBUG }, 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:46:31 DEBUG { 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:46:31 DEBUG "metric": { 18:46:31 DEBUG { 18:46:31 DEBUG "pod": "ds-idrepo-1" 18:46:31 DEBUG { 18:46:31 DEBUG }, 18:46:31 DEBUG }, 18:46:31 DEBUG "value": [ 18:46:31 DEBUG { 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "pod": "ds-idrepo-1" 18:46:31 DEBUG }, 18:46:31 DEBUG "instance": "10.142.0.107:9100" 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "instance": "10.142.0.107:9100" 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG }, 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 18:46:31 DEBUG }, 18:46:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:46:31 DEBUG "metric": { 18:46:31 DEBUG }, 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 18:46:31 DEBUG "metric": { 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG }, 18:46:31 DEBUG "value": [ 18:46:31 DEBUG }, 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "metric": { 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG "value": [ 18:46:31 DEBUG }, 18:46:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "pod": "ds-idrepo-1" 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG "pod": "ds-idrepo-1" 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG [...] 18:46:31 DEBUG [...] 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "pod": "ds-idrepo-1" 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG }, 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "pod": "ds-idrepo-0" 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG "instance": "10.142.0.43:9100" 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "instance": "10.142.0.43:9100" 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "instance": "10.142.0.43:9100" 18:46:31 DEBUG }, 18:46:31 DEBUG "instance": "10.142.0.43:9100" 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG }, 18:46:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG [...] 18:46:31 DEBUG "pod": "ds-cts-2" 18:46:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "pod": "ds-idrepo-0" 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG }, 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG }, 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG }, 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "0" 18:46:31 DEBUG 18:46:31 DEBUG [...] 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:46:31 DEBUG 18:46:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 18:46:31 DEBUG "pod": "ds-cts-2" 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "instance": "10.142.0.43:9100" 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG "5440.433333333333" 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG ] 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG }, 18:46:31 DEBUG "pod": "ds-idrepo-0" 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG 18:46:31 DEBUG }, 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG }, 18:46:31 DEBUG [...] 18:46:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:46:31 DEBUG 18:46:31 DEBUG ] 18:46:31 DEBUG "0" 18:46:31 DEBUG } 18:46:31 DEBUG "13763437568" 18:46:31 DEBUG 18:46:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:46:31 DEBUG "0" 18:46:31 DEBUG "value": [ 18:46:31 DEBUG }, 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG "0" 18:46:31 DEBUG "0.000018668888888888888" 18:46:31 DEBUG "2286.9" 18:46:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:46:31 DEBUG }, 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG "0" 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:46:31 DEBUG "0" 18:46:31 DEBUG "value": [ 18:46:31 DEBUG 18:46:31 DEBUG }, 18:46:31 DEBUG 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG 18:46:31 DEBUG }, 18:46:31 DEBUG ] 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "pod": "ds-idrepo-0" 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "6481.533333333333" 18:46:31 DEBUG ] 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG }, 18:46:31 DEBUG ] 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "pod": "ds-cts-2" 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG "pod": "ds-idrepo-0" 18:46:31 DEBUG "value": [ 18:46:31 DEBUG } 18:46:31 DEBUG "0.076" 18:46:31 DEBUG }, 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG }, 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG "0" 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG "0.00020000000000000757" 18:46:31 DEBUG "value": [ 18:46:31 DEBUG } 18:46:31 DEBUG "1270140928" 18:46:31 DEBUG 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG }, 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG }, 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "0" 18:46:31 DEBUG "value": [ 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG "2287" 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG "pod": "ds-cts-2" 18:46:31 DEBUG "0" 18:46:31 DEBUG "value": [ 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG "value": [ 18:46:31 DEBUG "6.763988275500126" 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG "124063" 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG }, 18:46:31 DEBUG ] 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG "2286.766666666667" 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG "value": [ 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG "0" 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG "0" 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG 1713203191, 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG "34589877.05648274" 18:46:31 DEBUG "0" 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG ] 18:46:31 DEBUG ] 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 DEBUG } 18:46:31 INFO [loop_until]: OK (rc = 0) 18:46:31 DEBUG --- stdout --- 18:46:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5131Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4575m 15% 10520Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 9866m 33% 10388Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17449m 58% 3097Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1836m 6% 7127Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 6668m 22% 12539Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2997Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 9623m 32% 9964Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1768m 5% 6267Mi 5% gke-xlou-cdm-ds-0f601751-j488 4608m 15% 10055Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2452Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 13162m 44% 10279Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 11226m 37% 6908Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6849m 43% 4995Mi 8% 18:46:31 DEBUG --- stderr --- 18:46:31 DEBUG 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 INFO 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 INFO [loop_until]: kubectl --namespace=xlou top pods 18:47:31 DEBUG --- status code --- 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203251" 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG --- status code --- 18:47:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 ERROR [http_cmd]: ERROR 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG --- http response --- 18:47:31 INFO 18:47:31 DEBUG --- http response --- 18:47:31 ERROR --- status code --- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG --- http response --- 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:47:31 INFO [loop_until]: kubectl --namespace=xlou top node 18:47:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:47:31 DEBUG --- status code --- 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG ----- output ----- 18:47:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:47:31 ERROR --- http response --- 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 DEBUG --- status code --- 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 DEBUG { 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203251" 18:47:31 DEBUG { 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:47:31 DEBUG { 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 DEBUG { 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203251" 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG --- status code --- 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 DEBUG "status": "success", 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203251" 18:47:31 DEBUG "status": "success", 18:47:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG ----- output ----- 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG "data": { 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 DEBUG "data": { 18:47:31 DEBUG "data": { 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG "data": { 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG --- http response --- 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG { 18:47:31 DEBUG "result": [ 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG { 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG "result": [ 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG "result": [ 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG { 18:47:31 DEBUG "result": [ 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG ----- output ----- 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG { 18:47:31 DEBUG "data": { 18:47:31 DEBUG "metric": { 18:47:31 DEBUG { 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:47:31 INFO [http_cmd]: http status code OK 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:47:31 DEBUG "data": { 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:47:31 DEBUG { 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG { 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG "metric": { 18:47:31 DEBUG { 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:47:31 DEBUG "metric": { 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:47:31 DEBUG "metric": { 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG --- status code --- 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG "data": { 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:47:31 DEBUG "pod": "ds-idrepo-1" 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG "pod": "ds-idrepo-1" 18:47:31 DEBUG "result": [ 18:47:31 DEBUG }, 18:47:31 DEBUG "instance": "10.142.0.107:9100" 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:47:31 DEBUG { 18:47:31 DEBUG "result": [ 18:47:31 DEBUG { 18:47:31 DEBUG "data": { 18:47:31 DEBUG { 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG "data": { 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG }, 18:47:31 DEBUG "data": { 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG }, 18:47:31 DEBUG { 18:47:31 DEBUG "value": [ 18:47:31 DEBUG }, 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG --- http response --- 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG { 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG "result": [ 18:47:31 DEBUG { 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "metric": { 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "data": { 18:47:31 DEBUG "data": { 18:47:31 DEBUG "data": { 18:47:31 DEBUG "data": { 18:47:31 DEBUG "data": { 18:47:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:47:31 DEBUG "data": { 18:47:31 DEBUG "metric": { 18:47:31 DEBUG "data": { 18:47:31 DEBUG "result": [ 18:47:31 DEBUG "data": { 18:47:31 DEBUG "result": [ 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG 18:47:31 DEBUG "result": [ 18:47:31 DEBUG "data": { 18:47:31 DEBUG "data": { 18:47:31 DEBUG "data": { 18:47:31 DEBUG 18:47:31 DEBUG "pod": "ds-idrepo-1" 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG ----- output ----- 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG { 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG "metric": { 18:47:31 DEBUG "data": { 18:47:31 DEBUG 18:47:31 DEBUG { 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "result": [ 18:47:31 DEBUG "result": [ 18:47:31 DEBUG "result": [ 18:47:31 DEBUG "result": [ 18:47:31 DEBUG "result": [ 18:47:31 DEBUG { 18:47:31 DEBUG "result": [ 18:47:31 DEBUG }, 18:47:31 DEBUG "result": [ 18:47:31 DEBUG "metric": { 18:47:31 DEBUG "result": [ 18:47:31 DEBUG "metric": { 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG "data": { 18:47:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG 18:47:31 DEBUG "metric": { 18:47:31 DEBUG "result": [ 18:47:31 DEBUG "result": [ 18:47:31 DEBUG "result": [ 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG [...] 18:47:31 DEBUG 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG "status": "success", 18:47:31 DEBUG { 18:47:31 DEBUG "value": [ 18:47:31 DEBUG { 18:47:31 DEBUG "pod": "ds-idrepo-1" 18:47:31 DEBUG { 18:47:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:47:31 DEBUG "data": { 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG }, 18:47:31 DEBUG "result": [ 18:47:31 DEBUG [...] 18:47:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG { 18:47:31 DEBUG [...] 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG [...] 18:47:31 DEBUG "metric": { 18:47:31 DEBUG "metric": { 18:47:31 DEBUG "metric": { 18:47:31 DEBUG "metric": { 18:47:31 DEBUG "metric": { 18:47:31 DEBUG "data": { 18:47:31 DEBUG "metric": { 18:47:31 DEBUG 18:47:31 DEBUG "metric": { 18:47:31 DEBUG }, 18:47:31 DEBUG "metric": { 18:47:31 DEBUG }, 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG "result": [ 18:47:31 DEBUG "value": [ 18:47:31 DEBUG { 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG "metric": { 18:47:31 DEBUG "metric": { 18:47:31 DEBUG "metric": { 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:47:31 DEBUG "pod": "ds-idrepo-1" 18:47:31 DEBUG "instance": "10.142.0.107:9100" 18:47:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:47:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:47:31 DEBUG "resultType": "vector", 18:47:31 DEBUG "pod": "ds-idrepo-1" 18:47:31 DEBUG 18:47:31 INFO [loop_until]: OK (rc = 0) 18:47:31 DEBUG "instance": "10.142.0.107:9100" 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "instance": "10.142.0.107:9100" 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "result": [ 18:47:31 DEBUG { 18:47:31 DEBUG 18:47:31 DEBUG "metric": { 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:47:31 DEBUG "instance": "10.142.0.107:9100" 18:47:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG }, 18:47:31 DEBUG }, 18:47:31 DEBUG }, 18:47:31 DEBUG }, 18:47:31 DEBUG "result": [ 18:47:31 DEBUG }, 18:47:31 DEBUG 18:47:31 DEBUG --- stdout --- 18:47:31 DEBUG }, 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG 18:47:31 DEBUG { 18:47:31 DEBUG "metric": { 18:47:31 DEBUG 18:47:31 DEBUG "pod": "ds-idrepo-1" 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG }, 18:47:31 DEBUG }, 18:47:31 DEBUG 18:47:31 DEBUG [...] 18:47:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "value": [ 18:47:31 DEBUG { 18:47:31 DEBUG "value": [ 18:47:31 DEBUG [...] 18:47:31 DEBUG "value": [ 18:47:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 9919m 8334Mi am-7d597cf7d6-8vr8f 8978m 8446Mi am-7d597cf7d6-9wz9h 12345m 8650Mi ds-cts-0 6234m 8175Mi ds-cts-1 4595m 8353Mi ds-cts-2 4483m 8503Mi ds-idrepo-0 11950m 4802Mi ds-idrepo-1 1780m 4340Mi ds-idrepo-2 1685m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-7c47b75958-dgfm9 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6646m 1123Mi 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG 18:47:31 DEBUG "metric": { 18:47:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG "value": [ 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "pod": "ds-idrepo-0" 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG "instance": "10.142.0.43:9100" 18:47:31 DEBUG "value": [ 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "metric": { 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG --- stderr --- 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:47:31 DEBUG }, 18:47:31 DEBUG [...] 18:47:31 DEBUG "pod": "ds-cts-2" 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG }, 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG [...] 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG [...] 18:47:31 DEBUG }, 18:47:31 DEBUG "value": [ 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG 18:47:31 DEBUG [...] 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG "value": [ 18:47:31 DEBUG 18:47:31 DEBUG [...] 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG [...] 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG [...] 18:47:31 DEBUG "pod": "ds-cts-2" 18:47:31 DEBUG "2308.9333333333334" 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG 18:47:31 DEBUG [...] 18:47:31 DEBUG [...] 18:47:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:47:31 DEBUG [...] 18:47:31 DEBUG 18:47:31 DEBUG [...] 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG [...] 18:47:31 DEBUG 18:47:31 DEBUG [...] 18:47:31 DEBUG [...] 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG ] 18:47:31 DEBUG "0.0003333333333330302" 18:47:31 DEBUG [...] 18:47:31 DEBUG [...] 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "pod": "ds-cts-2" 18:47:31 DEBUG "0.000018668888888888888" 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "0" 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG } 18:47:31 DEBUG ] 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG 18:47:31 DEBUG "pod": "ds-idrepo-0" 18:47:31 DEBUG 18:47:31 DEBUG "pod": "ds-cts-2" 18:47:31 DEBUG 18:47:31 DEBUG [...] 18:47:31 DEBUG }, 18:47:31 DEBUG ] 18:47:31 DEBUG 18:47:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG ] 18:47:31 DEBUG 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG ] 18:47:31 DEBUG } 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "pod": "ds-idrepo-0" 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG [...] 18:47:31 DEBUG 18:47:31 DEBUG "value": [ 18:47:31 DEBUG } 18:47:31 DEBUG 18:47:31 DEBUG }, 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG } 18:47:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:47:31 DEBUG "0" 18:47:31 DEBUG } 18:47:31 DEBUG ] 18:47:31 DEBUG 18:47:31 DEBUG "instance": "10.142.0.43:9100" 18:47:31 DEBUG 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG }, 18:47:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:47:31 DEBUG "pod": "ds-idrepo-0" 18:47:31 DEBUG "0" 18:47:31 DEBUG "instance": "10.142.0.43:9100" 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "instance": "10.142.0.43:9100" 18:47:31 DEBUG "value": [ 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG ] 18:47:31 DEBUG "pod": "ds-idrepo-0" 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:47:31 DEBUG "instance": "10.142.0.43:9100" 18:47:31 DEBUG ] 18:47:31 DEBUG }, 18:47:31 DEBUG ] 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG 18:47:31 DEBUG "pod": "ds-cts-2" 18:47:31 DEBUG 18:47:31 DEBUG [...] 18:47:31 DEBUG "value": [ 18:47:31 DEBUG }, 18:47:31 DEBUG }, 18:47:31 DEBUG ] 18:47:31 DEBUG }, 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG }, 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG 18:47:31 DEBUG 18:47:31 DEBUG "684988.3053029528" 18:47:31 DEBUG ] 18:47:31 DEBUG }, 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG }, 18:47:31 DEBUG }, 18:47:31 DEBUG } 18:47:31 DEBUG "value": [ 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG }, 18:47:31 DEBUG }, 18:47:31 DEBUG "0" 18:47:31 DEBUG 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "value": [ 18:47:31 DEBUG } 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "0" 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "0" 18:47:31 DEBUG ] 18:47:31 DEBUG 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "6.73959149274541" 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "value": [ 18:47:31 DEBUG } 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG ] 18:47:31 DEBUG "value": [ 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG ] 18:47:31 DEBUG 18:47:31 DEBUG "0" 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG ] 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG ] 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG } 18:47:31 DEBUG "pod": "ds-cts-2" 18:47:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:47:31 DEBUG } 18:47:31 DEBUG ] 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG ] 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG "1272406016" 18:47:31 DEBUG } 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "5034.266666666666" 18:47:31 DEBUG } 18:47:31 DEBUG 18:47:31 DEBUG ] 18:47:31 DEBUG "262042" 18:47:31 DEBUG "0" 18:47:31 DEBUG } 18:47:31 DEBUG "13763970048" 18:47:31 DEBUG } 18:47:31 DEBUG "0" 18:47:31 DEBUG ] 18:47:31 DEBUG }, 18:47:31 DEBUG }, 18:47:31 DEBUG } 18:47:31 DEBUG "0" 18:47:31 DEBUG } 18:47:31 DEBUG "0.076" 18:47:31 DEBUG "5132.700000000001" 18:47:31 DEBUG ] 18:47:31 DEBUG } 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG ] 18:47:31 DEBUG ] 18:47:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:47:31 DEBUG } 18:47:31 DEBUG ] 18:47:31 DEBUG ] 18:47:31 DEBUG } 18:47:31 DEBUG ] 18:47:31 DEBUG ] 18:47:31 DEBUG ] 18:47:31 DEBUG } 18:47:31 DEBUG "value": [ 18:47:31 DEBUG "value": [ 18:47:31 DEBUG } 18:47:31 DEBUG ] 18:47:31 DEBUG ] 18:47:31 DEBUG ] 18:47:31 DEBUG ] 18:47:31 DEBUG } 18:47:31 DEBUG "23258490.316051904" 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG }, 18:47:31 DEBUG ] 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG ] 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG 1713203251, 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG ] 18:47:31 DEBUG ] 18:47:31 DEBUG ] 18:47:31 DEBUG } 18:47:31 DEBUG "value": [ 18:47:31 DEBUG } 18:47:31 DEBUG ] 18:47:31 DEBUG ] 18:47:31 DEBUG ] 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG "2.7647298176764887" 18:47:31 DEBUG "2308.866666666667" 18:47:31 DEBUG ] 18:47:31 DEBUG } 18:47:31 DEBUG ] 18:47:31 DEBUG ] 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG 1713203251, 18:47:31 INFO [loop_until]: OK (rc = 0) 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG ] 18:47:31 DEBUG ] 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG ] 18:47:31 DEBUG } 18:47:31 DEBUG "2309.0333333333333" 18:47:31 DEBUG --- stdout --- 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG ] 18:47:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5134Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4604m 15% 10550Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10206m 34% 10382Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17596m 58% 3084Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1866m 6% 7121Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 6806m 22% 12525Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2974Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 9760m 32% 9961Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1786m 5% 6253Mi 5% gke-xlou-cdm-ds-0f601751-j488 4549m 15% 10067Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2436Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 13226m 44% 10282Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 11507m 38% 6949Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6898m 43% 4999Mi 8% 18:47:31 DEBUG ] 18:47:31 DEBUG ] 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG --- stderr --- 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG ] 18:47:31 DEBUG 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:47:31 DEBUG } 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203311" 18:48:31 INFO 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203311" 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 ERROR [http_cmd]: ERROR 18:48:31 INFO [loop_until]: kubectl --namespace=xlou top node 18:48:31 DEBUG --- status code --- 18:48:31 ERROR --- status code --- 18:48:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:48:31 DEBUG --- http response --- 18:48:31 ERROR --- http response --- 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:48:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:48:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:48:31 INFO 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [loop_until]: kubectl --namespace=xlou top pods 18:48:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:48:31 DEBUG --- status code --- 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 DEBUG --- status code --- 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:48:31 DEBUG { 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG { 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG { 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG { 18:48:31 DEBUG "data": { 18:48:31 DEBUG { 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG "data": { 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG "data": { 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "data": { 18:48:31 DEBUG "result": [ 18:48:31 DEBUG "data": { 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "result": [ 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG { 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "result": [ 18:48:31 DEBUG { 18:48:31 DEBUG "result": [ 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "result": [ 18:48:31 DEBUG { 18:48:31 DEBUG "metric": { 18:48:31 DEBUG { 18:48:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:48:31 DEBUG { 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:48:31 DEBUG "metric": { 18:48:31 DEBUG }, 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "instance": "10.142.0.107:9100" 18:48:31 DEBUG }, 18:48:31 DEBUG "instance": "10.142.0.107:9100" 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "pod": "ds-idrepo-1" 18:48:31 DEBUG }, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG }, 18:48:31 DEBUG 18:48:31 DEBUG }, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG [...] 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG [...] 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG [...] 18:48:31 DEBUG 18:48:31 DEBUG [...] 18:48:31 DEBUG 18:48:31 DEBUG [...] 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:48:31 DEBUG 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 DEBUG }, 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG 18:48:31 DEBUG "instance": "10.142.0.43:9100" 18:48:31 DEBUG }, 18:48:31 DEBUG "instance": "10.142.0.43:9100" 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG "value": [ 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG "pod": "ds-cts-2" 18:48:31 DEBUG }, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG }, 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG }, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG "733.4333333333333" 18:48:31 DEBUG "value": [ 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG "2303" 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG ] 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG "13764598784" 18:48:31 DEBUG ] 18:48:31 DEBUG "6881.599999999999" 18:48:31 DEBUG } 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG "0" 18:48:31 DEBUG { 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG { 18:48:31 DEBUG ] 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG } 18:48:31 DEBUG "data": { 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG "data": { 18:48:31 DEBUG ] 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG } 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG } 18:48:31 DEBUG "result": [ 18:48:31 DEBUG } 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG } 18:48:31 DEBUG "result": [ 18:48:31 DEBUG } 18:48:31 DEBUG { 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG { 18:48:31 DEBUG "metric": { 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "pod": "ds-idrepo-1" 18:48:31 DEBUG "pod": "ds-cts-1" 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:48:31 DEBUG }, 18:48:31 DEBUG }, 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "value": [ 18:48:31 DEBUG { 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "data": { 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG [...] 18:48:31 DEBUG [...] 18:48:31 DEBUG "result": [ 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG { 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "metric": { 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "instance": "10.142.0.107:9100" 18:48:31 DEBUG "pod": "ds-idrepo-0" 18:48:31 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:48:31 DEBUG }, 18:48:31 DEBUG }, 18:48:31 DEBUG }, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG 18:48:31 DEBUG "0" 18:48:31 DEBUG "0.005064602706770033" 18:48:31 DEBUG 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG [...] 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG "instance": "10.142.0.43:9100" 18:48:31 DEBUG }, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG "0.00033333333333333365" 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG { 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG "data": { 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "result": [ 18:48:31 DEBUG { 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "pod": "ds-cts-1" 18:48:31 DEBUG }, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG [...] 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:48:31 DEBUG }, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG "0" 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203311" 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203311" 18:48:31 DEBUG --- status code --- 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203311" 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG --- http response --- 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 DEBUG --- status code --- 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG --- status code --- 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 INFO [loop_until]: OK (rc = 0) 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG { 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:48:31 DEBUG --- http response --- 18:48:31 INFO [loop_until]: OK (rc = 0) 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG --- stdout --- 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:48:31 INFO [http_cmd]: http status code OK 18:48:31 DEBUG --- stdout --- 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:48:31 DEBUG { 18:48:31 DEBUG { 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:48:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 68m 0% 5143Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 6839m 22% 10514Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11456m 38% 10402Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17535m 58% 3094Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1909m 6% 7119Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 9418m 31% 12485Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2978Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10779m 36% 9969Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1848m 6% 6271Mi 5% gke-xlou-cdm-ds-0f601751-j488 7654m 25% 10033Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2437Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 14568m 48% 10318Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12249m 41% 7002Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6860m 43% 5024Mi 8% 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG "data": { 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG { 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG --- status code --- 18:48:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 11009m 8343Mi am-7d597cf7d6-8vr8f 9504m 8452Mi am-7d597cf7d6-9wz9h 13648m 8681Mi ds-cts-0 8178m 8179Mi ds-cts-1 6905m 8358Mi ds-cts-2 5805m 8509Mi ds-idrepo-0 11624m 4862Mi ds-idrepo-1 1880m 4340Mi ds-idrepo-2 1785m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1022Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-7c47b75958-dgfm9 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7053m 1152Mi 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG { 18:48:31 DEBUG { 18:48:31 DEBUG { 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG --- stderr --- 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG { 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG { 18:48:31 DEBUG { 18:48:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:48:31 DEBUG --- stderr --- 18:48:31 DEBUG { 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG { 18:48:31 DEBUG "data": { 18:48:31 DEBUG "data": { 18:48:31 DEBUG { 18:48:31 DEBUG 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 18:48:31 DEBUG "result": [ 18:48:31 DEBUG { 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG "data": { 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG --- http response --- 18:48:31 DEBUG 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG "data": { 18:48:31 DEBUG "data": { 18:48:31 DEBUG "data": { 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG { 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG "data": { 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "data": { 18:48:31 DEBUG "data": { 18:48:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:48:31 DEBUG "data": { 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "data": { 18:48:31 DEBUG "result": [ 18:48:31 DEBUG "result": [ 18:48:31 DEBUG "data": { 18:48:31 DEBUG { 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "data": { 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "result": [ 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG ----- output ----- 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "result": [ 18:48:31 DEBUG "result": [ 18:48:31 DEBUG "result": [ 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG { 18:48:31 DEBUG { 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "result": [ 18:48:31 DEBUG { 18:48:31 DEBUG "result": [ 18:48:31 DEBUG "result": [ 18:48:31 DEBUG { 18:48:31 DEBUG "result": [ 18:48:31 DEBUG { 18:48:31 DEBUG { 18:48:31 DEBUG { 18:48:31 DEBUG "result": [ 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "result": [ 18:48:31 DEBUG "data": { 18:48:31 DEBUG }, 18:48:31 DEBUG "result": [ 18:48:31 DEBUG { 18:48:31 DEBUG "metric": { 18:48:31 DEBUG { 18:48:31 DEBUG { 18:48:31 DEBUG "status": "success", 18:48:31 DEBUG { 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "metric": { 18:48:31 DEBUG { 18:48:31 DEBUG "pod": "ds-cts-1" 18:48:31 DEBUG "pod": "ds-idrepo-1" 18:48:31 DEBUG { 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "value": [ 18:48:31 DEBUG { 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "pod": "ds-idrepo-1" 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "data": { 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "instance": "10.142.0.107:9100" 18:48:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:48:31 DEBUG "instance": "10.142.0.107:9100" 18:48:31 DEBUG "metric": { 18:48:31 DEBUG }, 18:48:31 DEBUG }, 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "result": [ 18:48:31 DEBUG 18:48:31 DEBUG "metric": { 18:48:31 DEBUG "pod": "ds-idrepo-1" 18:48:31 DEBUG }, 18:48:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:48:31 DEBUG "pod": "ds-idrepo-1" 18:48:31 DEBUG "resultType": "vector", 18:48:31 DEBUG "pod": "ds-idrepo-1" 18:48:31 DEBUG }, 18:48:31 DEBUG }, 18:48:31 DEBUG }, 18:48:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:48:31 DEBUG { 18:48:31 DEBUG 18:48:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:48:31 DEBUG }, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG }, 18:48:31 DEBUG }, 18:48:31 DEBUG "result": [ 18:48:31 DEBUG }, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "value": [ 18:48:31 DEBUG }, 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG }, 18:48:31 DEBUG "metric": { 18:48:31 DEBUG 18:48:31 DEBUG }, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "value": [ 18:48:31 DEBUG { 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "pod": "ds-cts-1" 18:48:31 DEBUG [...] 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "metric": { 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG }, 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG [...] 18:48:31 DEBUG [...] 18:48:31 DEBUG 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG [...] 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG }, 18:48:31 DEBUG 18:48:31 DEBUG [...] 18:48:31 DEBUG [...] 18:48:31 DEBUG [...] 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG [...] 18:48:31 DEBUG 18:48:31 DEBUG [...] 18:48:31 DEBUG [...] 18:48:31 DEBUG "value": [ 18:48:31 DEBUG [...] 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG [...] 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG [...] 18:48:31 DEBUG 18:48:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:48:31 DEBUG [...] 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG }, 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:48:31 DEBUG "pod": "ds-cts-2" 18:48:31 DEBUG 18:48:31 DEBUG [...] 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "pod": "ds-idrepo-0" 18:48:31 DEBUG 18:48:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG "instance": "10.142.0.43:9100" 18:48:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:48:31 DEBUG "instance": "10.142.0.43:9100" 18:48:31 DEBUG 18:48:31 DEBUG }, 18:48:31 DEBUG }, 18:48:31 DEBUG 18:48:31 DEBUG 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG 18:48:31 DEBUG "pod": "ds-idrepo-0" 18:48:31 DEBUG }, 18:48:31 DEBUG 18:48:31 DEBUG }, 18:48:31 DEBUG [...] 18:48:31 DEBUG "pod": "ds-idrepo-0" 18:48:31 DEBUG }, 18:48:31 DEBUG }, 18:48:31 DEBUG }, 18:48:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:48:31 DEBUG 18:48:31 DEBUG "0" 18:48:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:48:31 DEBUG }, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "pod": "ds-idrepo-0" 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 18:48:31 DEBUG }, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "value": [ 18:48:31 DEBUG }, 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG }, 18:48:31 DEBUG 18:48:31 DEBUG ] 18:48:31 DEBUG }, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG }, 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "72319249.37716314" 18:48:31 DEBUG "0" 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:48:31 DEBUG } 18:48:31 DEBUG "value": [ 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG "0" 18:48:31 DEBUG "value": [ 18:48:31 DEBUG "388876" 18:48:31 DEBUG 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG "0.00040000000000001514" 18:48:31 DEBUG "6.727520167856923" 18:48:31 DEBUG "5318.6" 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG }, 18:48:31 DEBUG ] 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG "0" 18:48:31 DEBUG ] 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG ] 18:48:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:48:31 DEBUG "0" 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG "1301745664" 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG "2303.1" 18:48:31 DEBUG "value": [ 18:48:31 DEBUG } 18:48:31 DEBUG "2303.133333333333" 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG "0" 18:48:31 DEBUG } 18:48:31 DEBUG }, 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG "value": [ 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG "1102.0875199191005" 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG 1713203311, 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG "0.076" 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG } 18:48:31 DEBUG ] 18:48:31 DEBUG } 18:48:31 DEBUG } 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "data": { 18:49:31 DEBUG "data": { 18:49:31 DEBUG "data": { 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "result": [ 18:49:31 DEBUG "result": [ 18:49:31 DEBUG "result": [ 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "pod": "ds-idrepo-1" 18:49:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:49:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:49:31 DEBUG }, 18:49:31 DEBUG }, 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG "value": [ 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG [...] 18:49:31 DEBUG [...] 18:49:31 DEBUG [...] 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "pod": "ds-idrepo-0" 18:49:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:49:31 DEBUG "pod": "ds-cts-2" 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG }, 18:49:31 DEBUG }, 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG "value": [ 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG "0" 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG 1713203371, 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG ] 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG "10.138931346200833" 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG "2287.4666666666667" 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG } 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG ] 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG ] 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG ] 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG } 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:49:31 ERROR [http_cmd]: ERROR 18:49:31 DEBUG ] 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG ] 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:49:31 DEBUG } 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG ----- output ----- 18:49:31 ERROR --- status code --- 18:49:31 DEBUG } 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG } 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:49:31 DEBUG } 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG } 18:49:31 DEBUG { 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "status": "success", 18:49:31 ERROR --- http response --- 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG "data": { 18:49:31 DEBUG "data": { 18:49:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:49:31 DEBUG "data": { 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:49:31 DEBUG "resultType": "vector", 18:49:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG "result": [ 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG "result": [ 18:49:31 DEBUG "result": [ 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "data": { 18:49:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:49:31 DEBUG "data": { 18:49:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:49:31 DEBUG "instance": "10.142.0.107:9100" 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG }, 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG }, 18:49:31 DEBUG }, 18:49:31 DEBUG "result": [ 18:49:31 DEBUG "value": [ 18:49:31 DEBUG "result": [ 18:49:31 DEBUG "value": [ 18:49:31 DEBUG "value": [ 18:49:31 DEBUG { 18:49:31 DEBUG 18:49:31 DEBUG { 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "metric": { 18:49:31 DEBUG 18:49:31 DEBUG "metric": { 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "pod": "ds-idrepo-1" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203371" 18:49:31 DEBUG 18:49:31 DEBUG "instance": "10.142.0.107:9100" 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG }, 18:49:31 DEBUG [...] 18:49:31 DEBUG }, 18:49:31 DEBUG [...] 18:49:31 DEBUG [...] 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:49:31 DEBUG 18:49:31 DEBUG "pod": "ds-cts-2" 18:49:31 DEBUG "instance": "10.142.0.43:9100" 18:49:31 DEBUG [...] 18:49:31 DEBUG }, 18:49:31 DEBUG [...] 18:49:31 DEBUG }, 18:49:31 DEBUG }, 18:49:31 DEBUG 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG "value": [ 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG 18:49:31 DEBUG "2282.2" 18:49:31 DEBUG 18:49:31 DEBUG "142.96185124428467" 18:49:31 DEBUG "13765198848" 18:49:31 DEBUG "pod": "ds-cts-2" 18:49:31 DEBUG ] 18:49:31 DEBUG "instance": "10.142.0.43:9100" 18:49:31 DEBUG ] 18:49:31 DEBUG ] 18:49:31 DEBUG }, 18:49:31 DEBUG } 18:49:31 DEBUG }, 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG "value": [ 18:49:31 DEBUG ] 18:49:31 DEBUG "value": [ 18:49:31 DEBUG ] 18:49:31 DEBUG ] 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG } 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG "0" 18:49:31 DEBUG } 18:49:31 DEBUG "0" 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG { 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "data": { 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "result": [ 18:49:31 DEBUG { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "instance": "10.142.0.107:9100" 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG [...] 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "instance": "10.142.0.43:9100" 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG "5609.62762942961" 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG { 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "data": { 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "result": [ 18:49:31 DEBUG { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG [...] 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "pod": "ds-cts-2" 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG "39080051.65613766" 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG --- http response --- 18:49:31 INFO 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:49:31 INFO [loop_until]: kubectl --namespace=xlou top node 18:49:31 DEBUG ----- output ----- 18:49:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:49:31 INFO 18:49:31 DEBUG { 18:49:31 INFO [loop_until]: kubectl --namespace=xlou top pods 18:49:31 DEBUG "status": "success", 18:49:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:49:31 DEBUG "data": { 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "result": [ 18:49:31 DEBUG { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG [...] 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "pod": "ds-cts-2" 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG "117657.60357404628" 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG { 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "data": { 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "result": [ 18:49:31 DEBUG { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG [...] 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG "0" 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203371" 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG --- status code --- 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG --- status code --- 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG --- status code --- 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG { 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG { 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG { 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG { 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG { 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "data": { 18:49:31 INFO [http_cmd]: http status code OK 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "data": { 18:49:31 DEBUG "data": { 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:49:31 DEBUG "data": { 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "data": { 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG --- status code --- 18:49:31 DEBUG "data": { 18:49:31 DEBUG "data": { 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG "data": { 18:49:31 DEBUG "data": { 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "result": [ 18:49:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "result": [ 18:49:31 DEBUG "result": [ 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 DEBUG "result": [ 18:49:31 DEBUG { 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "result": [ 18:49:31 DEBUG { 18:49:31 DEBUG --- http response --- 18:49:31 DEBUG "result": [ 18:49:31 DEBUG "result": [ 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG { 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG "result": [ 18:49:31 DEBUG "result": [ 18:49:31 DEBUG { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG { 18:49:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:49:31 DEBUG { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "data": { 18:49:31 DEBUG "data": { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "data": { 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "pod": "ds-idrepo-1" 18:49:31 DEBUG "metric": { 18:49:31 DEBUG ----- output ----- 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "pod": "ds-idrepo-1" 18:49:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "pod": "ds-idrepo-1" 18:49:31 DEBUG }, 18:49:31 DEBUG "pod": "ds-idrepo-1" 18:49:31 DEBUG { 18:49:31 DEBUG "instance": "10.142.0.107:9100" 18:49:31 DEBUG }, 18:49:31 DEBUG }, 18:49:31 DEBUG "result": [ 18:49:31 DEBUG "result": [ 18:49:31 DEBUG }, 18:49:31 DEBUG "result": [ 18:49:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:49:31 DEBUG "pod": "ds-idrepo-1" 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG }, 18:49:31 DEBUG "status": "success", 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG "value": [ 18:49:31 DEBUG { 18:49:31 DEBUG { 18:49:31 DEBUG "value": [ 18:49:31 DEBUG { 18:49:31 DEBUG }, 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG "value": [ 18:49:31 DEBUG "data": { 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "metric": { 18:49:31 DEBUG 18:49:31 DEBUG "metric": { 18:49:31 DEBUG "value": [ 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "resultType": "vector", 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "instance": "10.142.0.107:9100" 18:49:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:49:31 DEBUG 18:49:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "result": [ 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG }, 18:49:31 DEBUG }, 18:49:31 DEBUG 18:49:31 DEBUG }, 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG [...] 18:49:31 DEBUG 18:49:31 DEBUG { 18:49:31 DEBUG 18:49:31 DEBUG [...] 18:49:31 DEBUG [...] 18:49:31 DEBUG "value": [ 18:49:31 DEBUG "value": [ 18:49:31 DEBUG [...] 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG [...] 18:49:31 DEBUG 18:49:31 DEBUG [...] 18:49:31 DEBUG "metric": { 18:49:31 DEBUG [...] 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG [...] 18:49:31 DEBUG [...] 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG }, 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "pod": "ds-idrepo-0" 18:49:31 DEBUG 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG "pod": "ds-idrepo-0" 18:49:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:49:31 DEBUG [...] 18:49:31 DEBUG [...] 18:49:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:49:31 DEBUG [...] 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "pod": "ds-cts-2" 18:49:31 DEBUG }, 18:49:31 DEBUG "pod": "ds-idrepo-0" 18:49:31 DEBUG 18:49:31 DEBUG "instance": "10.142.0.43:9100" 18:49:31 DEBUG }, 18:49:31 DEBUG }, 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG }, 18:49:31 DEBUG 18:49:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:49:31 DEBUG "pod": "ds-idrepo-0" 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG }, 18:49:31 DEBUG 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG }, 18:49:31 DEBUG }, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG 18:49:31 DEBUG 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG 18:49:31 DEBUG "value": [ 18:49:31 DEBUG "value": [ 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG "0" 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG [...] 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG "0" 18:49:31 DEBUG "2282.6" 18:49:31 DEBUG "instance": "10.142.0.43:9100" 18:49:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:49:31 DEBUG "413687.5" 18:49:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG "0" 18:49:31 DEBUG ] 18:49:31 DEBUG "0" 18:49:31 DEBUG 18:49:31 DEBUG "0.0004000400040004152" 18:49:31 DEBUG ] 18:49:31 DEBUG ] 18:49:31 DEBUG }, 18:49:31 DEBUG }, 18:49:31 DEBUG ] 18:49:31 DEBUG }, 18:49:31 DEBUG "1306578944" 18:49:31 DEBUG "0" 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG "value": [ 18:49:31 DEBUG "value": [ 18:49:31 DEBUG } 18:49:31 DEBUG "value": [ 18:49:31 DEBUG ] 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG ] 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG ] 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:49:31 DEBUG }, 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG "4850.5517218388495" 18:49:31 DEBUG "0.076" 18:49:31 DEBUG } 18:49:31 DEBUG "6.752135620670669" 18:49:31 DEBUG ] 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG "value": [ 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG 1713203371, 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG "2285.6" 18:49:31 DEBUG ] 18:49:31 DEBUG ] 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 DEBUG ] 18:49:31 DEBUG } 18:49:31 DEBUG } 18:49:31 INFO [loop_until]: OK (rc = 0) 18:49:31 DEBUG --- stdout --- 18:49:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5143Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10619m 35% 10494Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12869m 43% 11039Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 16957m 56% 3060Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1975m 6% 7121Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16200m 54% 12458Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2985Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12253m 41% 10504Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2037m 6% 6265Mi 5% gke-xlou-cdm-ds-0f601751-j488 10387m 34% 10025Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2444Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16670m 55% 10325Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12970m 43% 7013Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6874m 43% 5026Mi 8% 18:49:31 DEBUG --- stderr --- 18:49:31 DEBUG 18:49:31 INFO [loop_until]: OK (rc = 0) 18:49:31 DEBUG --- stdout --- 18:49:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12532m 8977Mi am-7d597cf7d6-8vr8f 11525m 8987Mi am-7d597cf7d6-9wz9h 15514m 8690Mi ds-cts-0 15313m 8174Mi ds-cts-1 10569m 8351Mi ds-cts-2 10543m 8504Mi ds-idrepo-0 12801m 4866Mi ds-idrepo-1 1967m 4340Mi ds-idrepo-2 1919m 4112Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1022Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-7c47b75958-dgfm9 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6593m 1153Mi 18:49:31 DEBUG --- stderr --- 18:49:31 DEBUG 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203431" 18:50:31 ERROR [http_cmd]: ERROR 18:50:31 ERROR --- status code --- 18:50:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:50:31 ERROR --- http response --- 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG --- status code --- 18:50:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "pod": "ds-cts-1" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "0.00002558398220246658" 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:50:31 DEBUG } 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:50:31 DEBUG }, 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "2294.6666666666665" 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203431" 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG { 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "result": [ 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG "metric": { 18:50:31 DEBUG { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG "pod": "ds-idrepo-1" 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG }, 18:50:31 DEBUG "pod": "ds-idrepo-1" 18:50:31 DEBUG }, 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG "value": [ 18:50:31 DEBUG }, 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:50:31 DEBUG "value": [ 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG { 18:50:31 DEBUG 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:50:31 DEBUG 18:50:31 DEBUG "data": { 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG { 18:50:31 DEBUG 18:50:31 DEBUG "result": [ 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "pod": "ds-idrepo-0" 18:50:31 DEBUG { 18:50:31 DEBUG 18:50:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:50:31 DEBUG "data": { 18:50:31 DEBUG }, 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "pod": "ds-cts-2" 18:50:31 DEBUG }, 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "value": [ 18:50:31 DEBUG "instance": "10.142.0.107:9100" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG "result": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG { 18:50:31 DEBUG "0" 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "2311.0666666666666" 18:50:31 DEBUG "metric": { 18:50:31 DEBUG ] 18:50:31 DEBUG 18:50:31 DEBUG "0.0007240822222222222" 18:50:31 DEBUG ] 18:50:31 DEBUG "instance": "10.142.0.107:9100" 18:50:31 DEBUG } 18:50:31 DEBUG 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG }, 18:50:31 DEBUG ] 18:50:31 DEBUG 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG "value": [ 18:50:31 DEBUG } 18:50:31 DEBUG [...] 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG 18:50:31 DEBUG } 18:50:31 DEBUG 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG } 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG "instance": "10.142.0.43:9100" 18:50:31 DEBUG 18:50:31 DEBUG }, 18:50:31 DEBUG 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "instance": "10.142.0.43:9100" 18:50:31 DEBUG "5121.4" 18:50:31 DEBUG }, 18:50:31 DEBUG ] 18:50:31 DEBUG "value": [ 18:50:31 DEBUG } 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG ] 18:50:31 DEBUG "0.00033333333333333365" 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG ] 18:50:31 DEBUG --- status code --- 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG } 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG } 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "instance": "10.142.0.107:9100" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG "instance": "10.142.0.43:9100" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "13765745664" 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "pod": "ds-cts-1" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "0" 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "pod": "ds-cts-1" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "0" 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "0" 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "pod": "ds-cts-1" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "348395.4022988506" 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "pod": "ds-idrepo-1" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG "pod": "ds-idrepo-0" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "0" 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "0.076" 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG { 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "data": { 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "result": [ 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 DEBUG { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "pod": "ds-idrepo-1" 18:50:31 DEBUG "pod": "ds-idrepo-1" 18:50:31 DEBUG }, 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG "pod": "ds-idrepo-0" 18:50:31 DEBUG "pod": "ds-idrepo-0" 18:50:31 DEBUG }, 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "0" 18:50:31 DEBUG "0" 18:50:31 DEBUG ] 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203431" 18:50:31 INFO 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203431" 18:50:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203431" 18:50:31 INFO [loop_until]: kubectl --namespace=xlou top node 18:50:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG "data": { 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG "resultType": "vector", 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG --- status code --- 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG "result": [ 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG { 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG "metric": { 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:50:31 DEBUG --- http response --- 18:50:31 INFO [http_cmd]: http status code OK 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:50:31 DEBUG "pod": "ds-idrepo-1" 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:50:31 DEBUG --- status code --- 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG }, 18:50:31 DEBUG { 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG { 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:50:31 DEBUG { 18:50:31 DEBUG "value": [ 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG { 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG { 18:50:31 DEBUG { 18:50:31 DEBUG --- http response --- 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG 18:50:31 DEBUG "data": { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG "data": { 18:50:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:50:31 DEBUG 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "data": { 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "data": { 18:50:31 DEBUG "data": { 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG ----- output ----- 18:50:31 DEBUG 18:50:31 DEBUG "result": [ 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "result": [ 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 DEBUG [...] 18:50:31 DEBUG { 18:50:31 DEBUG "result": [ 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 DEBUG "result": [ 18:50:31 DEBUG "result": [ 18:50:31 DEBUG { 18:50:31 DEBUG "status": "success", 18:50:31 DEBUG 18:50:31 DEBUG "metric": { 18:50:31 DEBUG { 18:50:31 DEBUG { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG { 18:50:31 DEBUG { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "data": { 18:50:31 DEBUG 18:50:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:50:31 DEBUG "resultType": "vector", 18:50:31 DEBUG 18:50:31 DEBUG }, 18:50:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:50:31 DEBUG "pod": "ds-idrepo-1" 18:50:31 DEBUG }, 18:50:31 DEBUG "instance": "10.142.0.107:9100" 18:50:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:50:31 DEBUG }, 18:50:31 DEBUG "result": [ 18:50:31 DEBUG "pod": "ds-idrepo-0" 18:50:31 DEBUG "value": [ 18:50:31 DEBUG }, 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG }, 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG { 18:50:31 DEBUG }, 18:50:31 DEBUG 18:50:31 DEBUG "value": [ 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG "value": [ 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG "metric": { 18:50:31 DEBUG "instance": "10.142.0.107:9100" 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG "0" 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG ] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG } 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG [...] 18:50:31 DEBUG ] 18:50:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:50:31 DEBUG 18:50:31 DEBUG 18:50:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:50:31 DEBUG "instance": "10.142.0.43:9100" 18:50:31 DEBUG 18:50:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:50:31 DEBUG 18:50:31 DEBUG } 18:50:31 DEBUG }, 18:50:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:50:31 DEBUG "pod": "ds-cts-2" 18:50:31 DEBUG }, 18:50:31 DEBUG }, 18:50:31 DEBUG 18:50:31 DEBUG }, 18:50:31 DEBUG 18:50:31 DEBUG } 18:50:31 DEBUG "value": [ 18:50:31 DEBUG }, 18:50:31 DEBUG }, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG "value": [ 18:50:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "value": [ 18:50:31 DEBUG "value": [ 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG }, 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "instance": "10.142.0.43:9100" 18:50:31 DEBUG "6.788950956376539" 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG "1616388096" 18:50:31 DEBUG "0.00033333333333350387" 18:50:31 DEBUG "value": [ 18:50:31 DEBUG "2311.5666666666666" 18:50:31 DEBUG }, 18:50:31 DEBUG ] 18:50:31 DEBUG "412217.5" 18:50:31 DEBUG "0" 18:50:31 DEBUG ] 18:50:31 DEBUG ] 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG ] 18:50:31 DEBUG "value": [ 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG "2279.5" 18:50:31 DEBUG } 18:50:31 DEBUG 1713203431, 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG ] 18:50:31 DEBUG ] 18:50:31 DEBUG ] 18:50:31 DEBUG "5120.133333333334" 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG ] 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 DEBUG } 18:50:31 INFO 18:50:31 INFO [loop_until]: kubectl --namespace=xlou top pods 18:50:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:50:31 INFO [loop_until]: OK (rc = 0) 18:50:31 DEBUG --- stdout --- 18:50:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5131Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10720m 35% 10508Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12979m 43% 11046Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17592m 58% 3070Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2042m 6% 7118Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16249m 54% 12466Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2979Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12408m 41% 10516Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2060m 6% 6264Mi 5% gke-xlou-cdm-ds-0f601751-j488 10419m 34% 10026Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2441Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16991m 56% 10471Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13006m 43% 7030Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6648m 41% 5321Mi 9% 18:50:31 DEBUG --- stderr --- 18:50:31 DEBUG 18:50:31 INFO [loop_until]: OK (rc = 0) 18:50:31 DEBUG --- stdout --- 18:50:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12575m 8981Mi am-7d597cf7d6-8vr8f 11847m 8998Mi am-7d597cf7d6-9wz9h 15978m 8816Mi ds-cts-0 15123m 8167Mi ds-cts-1 10828m 8345Mi ds-cts-2 10237m 8499Mi ds-idrepo-0 12594m 4881Mi ds-idrepo-1 1949m 4340Mi ds-idrepo-2 1875m 4112Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1022Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-7c47b75958-dgfm9 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6833m 1448Mi 18:50:31 DEBUG --- stderr --- 18:50:31 DEBUG 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- status code --- 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "data": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "2303.4666666666667" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG ----- output ----- 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG { 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG "data": { 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG "result": [ 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG { 18:51:31 DEBUG { 18:51:31 DEBUG { 18:51:31 DEBUG { 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG --- http response --- 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 ERROR [http_cmd]: ERROR 18:51:31 DEBUG "pod": "ds-idrepo-1" 18:51:31 DEBUG "data": { 18:51:31 DEBUG "data": { 18:51:31 DEBUG "data": { 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:51:31 DEBUG --- status code --- 18:51:31 ERROR --- status code --- 18:51:31 DEBUG }, 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:51:31 DEBUG "value": [ 18:51:31 DEBUG "result": [ 18:51:31 DEBUG "result": [ 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG --- http response --- 18:51:31 ERROR --- http response --- 18:51:31 DEBUG 18:51:31 DEBUG { 18:51:31 DEBUG { 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:51:31 DEBUG 18:51:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "data": { 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG 18:51:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:51:31 DEBUG "instance": "10.142.0.107:9100" 18:51:31 DEBUG "instance": "10.142.0.107:9100" 18:51:31 DEBUG "pod": "ds-idrepo-1" 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG { 18:51:31 DEBUG [...] 18:51:31 DEBUG }, 18:51:31 DEBUG }, 18:51:31 DEBUG }, 18:51:31 DEBUG "result": [ 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG 18:51:31 DEBUG "value": [ 18:51:31 DEBUG "value": [ 18:51:31 DEBUG "value": [ 18:51:31 DEBUG { 18:51:31 DEBUG "data": { 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:51:31 DEBUG "result": [ 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG "pod": "ds-cts-2" 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG }, 18:51:31 DEBUG { 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG }, 18:51:31 DEBUG [...] 18:51:31 DEBUG [...] 18:51:31 DEBUG [...] 18:51:31 DEBUG "value": [ 18:51:31 DEBUG "metric": { 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG }, 18:51:31 DEBUG "0.000018668888888888888" 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "value": [ 18:51:31 DEBUG ] 18:51:31 DEBUG "instance": "10.142.0.43:9100" 18:51:31 DEBUG "instance": "10.142.0.43:9100" 18:51:31 DEBUG "pod": "ds-idrepo-0" 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG } 18:51:31 DEBUG }, 18:51:31 DEBUG }, 18:51:31 DEBUG }, 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG ] 18:51:31 DEBUG "value": [ 18:51:31 DEBUG "value": [ 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG } 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG } 18:51:31 DEBUG "13766405120" 18:51:31 DEBUG "9983.466666666667" 18:51:31 DEBUG ] 18:51:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:51:31 DEBUG 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG "0" 18:51:31 DEBUG }, 18:51:31 DEBUG 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG ] 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG "0" 18:51:31 DEBUG }, 18:51:31 DEBUG } 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG "value": [ 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG { 18:51:31 DEBUG ] 18:51:31 DEBUG "2291.733333333333" 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG "data": { 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG ] 18:51:31 DEBUG "result": [ 18:51:31 DEBUG } 18:51:31 DEBUG { 18:51:31 DEBUG } 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "instance": "10.142.0.107:9100" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "instance": "10.142.0.43:9100" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "0" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "data": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "0" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "data": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "4666.430956590061" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "data": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "0.0765" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "data": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "pod": "ds-idrepo-1" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "pod": "ds-idrepo-0" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "0" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "data": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "0" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "data": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "pod": "ds-idrepo-1" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "pod": "ds-idrepo-0" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "0" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "data": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "0" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203491" 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "data": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "pod": "ds-idrepo-1" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "pod": "ds-idrepo-0" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "0" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "data": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "6.770625455958107" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "data": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "2278.7" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203491" 18:51:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203491" 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG --- http response --- 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG { 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "data": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "pod": "ds-idrepo-1" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "pod": "ds-idrepo-0" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "0" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "data": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "instance": "10.142.0.107:9100" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "instance": "10.142.0.43:9100" 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "0.0003333333333330302" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- http response --- 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG { 18:51:31 INFO [http_cmd]: http status code OK 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:51:31 DEBUG --- status code --- 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:51:31 DEBUG "data": { 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG { 18:51:31 DEBUG --- http response --- 18:51:31 DEBUG { 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:51:31 DEBUG { 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG "data": { 18:51:31 DEBUG "data": { 18:51:31 DEBUG ----- output ----- 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "data": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG { 18:51:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG "result": [ 18:51:31 DEBUG "result": [ 18:51:31 DEBUG "status": "success", 18:51:31 DEBUG }, 18:51:31 DEBUG "result": [ 18:51:31 DEBUG { 18:51:31 DEBUG { 18:51:31 DEBUG "data": { 18:51:31 DEBUG "value": [ 18:51:31 DEBUG { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "resultType": "vector", 18:51:31 DEBUG 18:51:31 DEBUG "metric": { 18:51:31 DEBUG "instance": "10.142.0.107:9100" 18:51:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:51:31 DEBUG "result": [ 18:51:31 DEBUG 18:51:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:51:31 DEBUG }, 18:51:31 DEBUG }, 18:51:31 DEBUG { 18:51:31 DEBUG 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG "value": [ 18:51:31 DEBUG "metric": { 18:51:31 DEBUG [...] 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "pod": "ds-idrepo-1" 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG }, 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:51:31 DEBUG [...] 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG }, 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG 18:51:31 DEBUG [...] 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG 18:51:31 DEBUG "instance": "10.142.0.43:9100" 18:51:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:51:31 DEBUG 18:51:31 DEBUG "1618296832" 18:51:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:51:31 DEBUG }, 18:51:31 DEBUG }, 18:51:31 DEBUG 18:51:31 DEBUG ] 18:51:31 DEBUG }, 18:51:31 DEBUG "value": [ 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 18:51:31 DEBUG } 18:51:31 DEBUG "value": [ 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "pod": "ds-cts-2" 18:51:31 DEBUG ] 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG "5951.633333333333" 18:51:31 DEBUG "412084" 18:51:31 DEBUG }, 18:51:31 DEBUG } 18:51:31 DEBUG "2291.0333333333333" 18:51:31 DEBUG ] 18:51:31 DEBUG ] 18:51:31 DEBUG "value": [ 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 DEBUG 1713203491, 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG ] 18:51:31 DEBUG "0" 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 DEBUG ] 18:51:31 DEBUG } 18:51:31 DEBUG } 18:51:31 INFO 18:51:31 INFO [loop_until]: kubectl --namespace=xlou top pods 18:51:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:51:31 INFO 18:51:31 INFO [loop_until]: kubectl --namespace=xlou top node 18:51:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:51:31 INFO [loop_until]: OK (rc = 0) 18:51:31 DEBUG --- stdout --- 18:51:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12659m 8969Mi am-7d597cf7d6-8vr8f 11833m 8998Mi am-7d597cf7d6-9wz9h 15695m 8809Mi ds-cts-0 15107m 8170Mi ds-cts-1 10624m 8350Mi ds-cts-2 10370m 8498Mi ds-idrepo-0 12805m 4895Mi ds-idrepo-1 1973m 4341Mi ds-idrepo-2 1931m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1022Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6617m 1450Mi 18:51:31 DEBUG --- stderr --- 18:51:31 DEBUG 18:51:31 INFO [loop_until]: OK (rc = 0) 18:51:31 DEBUG --- stdout --- 18:51:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5126Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10810m 36% 10506Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13036m 43% 11045Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17609m 58% 3071Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1984m 6% 7119Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 15929m 53% 12472Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2980Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12406m 41% 10520Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2063m 6% 6258Mi 5% gke-xlou-cdm-ds-0f601751-j488 10619m 35% 10036Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2453Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17100m 57% 10471Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12941m 43% 7045Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6842m 43% 5322Mi 9% 18:51:31 DEBUG --- stderr --- 18:51:31 DEBUG 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "2285.6666666666665" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203551" 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:52:31 DEBUG --- status code --- 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG { 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG "data": { 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:52:31 DEBUG { 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG "result": [ 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG { 18:52:31 DEBUG { 18:52:31 DEBUG { 18:52:31 DEBUG "data": { 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "status": "success", 18:52:31 ERROR [http_cmd]: ERROR 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG "data": { 18:52:31 DEBUG "pod": "ds-idrepo-1" 18:52:31 DEBUG "data": { 18:52:31 ERROR --- status code --- 18:52:31 DEBUG "result": [ 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG }, 18:52:31 DEBUG "resultType": "vector", 18:52:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:52:31 DEBUG { 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG "result": [ 18:52:31 DEBUG "value": [ 18:52:31 DEBUG "result": [ 18:52:31 ERROR --- http response --- 18:52:31 DEBUG "metric": { 18:52:31 DEBUG { 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:52:31 DEBUG 18:52:31 DEBUG { 18:52:31 DEBUG "instance": "10.142.0.107:9100" 18:52:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:52:31 DEBUG "metric": { 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG 18:52:31 DEBUG "metric": { 18:52:31 DEBUG }, 18:52:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:52:31 DEBUG "instance": "10.142.0.107:9100" 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG { 18:52:31 DEBUG 18:52:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:52:31 DEBUG "value": [ 18:52:31 DEBUG }, 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG [...] 18:52:31 DEBUG }, 18:52:31 DEBUG 18:52:31 DEBUG "value": [ 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG "data": { 18:52:31 DEBUG 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "result": [ 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG { 18:52:31 DEBUG { 18:52:31 DEBUG "pod": "ds-cts-2" 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG }, 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG 18:52:31 DEBUG "pod": "ds-idrepo-1" 18:52:31 DEBUG "data": { 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG 18:52:31 DEBUG }, 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG 18:52:31 DEBUG "instance": "10.142.0.43:9100" 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG 18:52:31 DEBUG "value": [ 18:52:31 DEBUG "result": [ 18:52:31 DEBUG "0" 18:52:31 DEBUG 18:52:31 DEBUG }, 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG "instance": "10.142.0.43:9100" 18:52:31 DEBUG 18:52:31 DEBUG { 18:52:31 DEBUG ] 18:52:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:52:31 DEBUG "value": [ 18:52:31 DEBUG }, 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:52:31 DEBUG 18:52:31 DEBUG "metric": { 18:52:31 DEBUG } 18:52:31 DEBUG }, 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG 18:52:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:52:31 DEBUG ] 18:52:31 DEBUG "value": [ 18:52:31 DEBUG "13766986752" 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG { 18:52:31 DEBUG [...] 18:52:31 DEBUG }, 18:52:31 DEBUG } 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG ] 18:52:31 DEBUG "0" 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG 18:52:31 DEBUG "value": [ 18:52:31 DEBUG } 18:52:31 DEBUG "11.156149823304968" 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG "data": { 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG ] 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG "result": [ 18:52:31 DEBUG "pod": "ds-idrepo-0" 18:52:31 DEBUG 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 DEBUG { 18:52:31 DEBUG }, 18:52:31 DEBUG [...] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG } 18:52:31 DEBUG "instance": "10.142.0.107:9100" 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG 18:52:31 DEBUG }, 18:52:31 DEBUG "0" 18:52:31 DEBUG 18:52:31 DEBUG "value": [ 18:52:31 DEBUG ] 18:52:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:52:31 DEBUG 18:52:31 DEBUG } 18:52:31 DEBUG }, 18:52:31 DEBUG 18:52:31 DEBUG ] 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG } 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG [...] 18:52:31 DEBUG } 18:52:31 DEBUG "2289" 18:52:31 DEBUG 18:52:31 DEBUG ] 18:52:31 DEBUG 18:52:31 DEBUG } 18:52:31 DEBUG 18:52:31 DEBUG ] 18:52:31 DEBUG "instance": "10.142.0.43:9100" 18:52:31 DEBUG } 18:52:31 DEBUG }, 18:52:31 DEBUG } 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "5313.633333333333" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "0" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "6723663.0201931195" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "0.077" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "285754280.9298506" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "pod": "ds-idrepo-1" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "pod": "ds-idrepo-0" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "0" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "0" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "6.697444654446586" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "pod": "ds-idrepo-1" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "pod": "ds-idrepo-0" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "0" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203551" 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "pod": "ds-idrepo-1" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "pod": "ds-idrepo-0" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "0" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "instance": "10.142.0.107:9100" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "instance": "10.142.0.43:9100" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "5497.466666666667" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "2289.4666666666667" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "pod": "ds-idrepo-1" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "pod": "ds-idrepo-0" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "0" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "2290.6" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203551" 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "instance": "10.142.0.107:9100" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "instance": "10.142.0.43:9100" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "0.00033333333333350387" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "412197.5" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "pod": "ds-idrepo-1" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "pod": "ds-cts-2" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "0" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO [http_cmd]: http status code OK 18:52:31 DEBUG --- status code --- 18:52:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:52:31 DEBUG --- http response --- 18:52:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:52:31 DEBUG ----- output ----- 18:52:31 DEBUG { 18:52:31 DEBUG "status": "success", 18:52:31 DEBUG "data": { 18:52:31 DEBUG "resultType": "vector", 18:52:31 DEBUG "result": [ 18:52:31 DEBUG { 18:52:31 DEBUG "metric": { 18:52:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG [...] 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG 18:52:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:52:31 DEBUG }, 18:52:31 DEBUG "value": [ 18:52:31 DEBUG 1713203551, 18:52:31 DEBUG "1615458304" 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG ] 18:52:31 DEBUG } 18:52:31 DEBUG } 18:52:31 INFO 18:52:31 INFO [loop_until]: kubectl --namespace=xlou top pods 18:52:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:52:31 INFO [loop_until]: OK (rc = 0) 18:52:31 DEBUG --- stdout --- 18:52:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12520m 8969Mi am-7d597cf7d6-8vr8f 11593m 8997Mi am-7d597cf7d6-9wz9h 15543m 8768Mi ds-cts-0 15380m 8189Mi ds-cts-1 10615m 8374Mi ds-cts-2 10552m 8519Mi ds-idrepo-0 12629m 4904Mi ds-idrepo-1 2038m 4341Mi ds-idrepo-2 1851m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1022Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6739m 1451Mi 18:52:31 DEBUG --- stderr --- 18:52:31 DEBUG 18:52:31 INFO 18:52:31 INFO [loop_until]: kubectl --namespace=xlou top node 18:52:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:52:31 INFO [loop_until]: OK (rc = 0) 18:52:31 DEBUG --- stdout --- 18:52:31 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5124Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10824m 36% 10518Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13007m 43% 11039Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17511m 58% 3067Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2017m 6% 7130Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16275m 54% 12476Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 61m 0% 2983Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12295m 41% 10522Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1944m 6% 6272Mi 5% gke-xlou-cdm-ds-0f601751-j488 10761m 36% 10038Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2439Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16885m 56% 10442Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12367m 41% 7056Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6815m 42% 5315Mi 9% 18:52:31 DEBUG --- stderr --- 18:52:31 DEBUG 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203611" 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "pod": "ds-idrepo-1" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "pod": "ds-cts-2" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "0.000054004444444444436" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 DEBUG ----- output ----- 18:53:31 ERROR [http_cmd]: ERROR 18:53:31 DEBUG { 18:53:31 ERROR --- status code --- 18:53:31 DEBUG "status": "success", 18:53:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:53:31 DEBUG "data": { 18:53:31 ERROR --- http response --- 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "instance": "10.142.0.107:9100" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "instance": "10.142.0.43:9100" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "13767654400" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:53:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "pod": "ds-idrepo-1" 18:53:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:53:31 DEBUG }, 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "pod": "ds-idrepo-0" 18:53:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:53:31 DEBUG }, 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "0" 18:53:31 DEBUG "2326.5666666666666" 18:53:31 DEBUG ] 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG } 18:53:31 DEBUG --- http response --- 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG { 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:53:31 DEBUG "data": { 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:53:31 DEBUG { 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG { 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "data": { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "data": { 18:53:31 DEBUG "result": [ 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG { 18:53:31 DEBUG "result": [ 18:53:31 DEBUG }, 18:53:31 DEBUG "result": [ 18:53:31 DEBUG "metric": { 18:53:31 DEBUG { 18:53:31 DEBUG "value": [ 18:53:31 DEBUG { 18:53:31 DEBUG "instance": "10.142.0.107:9100" 18:53:31 DEBUG "metric": { 18:53:31 DEBUG 18:53:31 DEBUG "metric": { 18:53:31 DEBUG }, 18:53:31 DEBUG "instance": "10.142.0.107:9100" 18:53:31 DEBUG 18:53:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:53:31 DEBUG "value": [ 18:53:31 DEBUG }, 18:53:31 DEBUG 18:53:31 DEBUG }, 18:53:31 DEBUG 18:53:31 DEBUG "value": [ 18:53:31 DEBUG [...] 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG }, 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG "instance": "10.142.0.43:9100" 18:53:31 DEBUG 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG 18:53:31 DEBUG }, 18:53:31 DEBUG "instance": "10.142.0.43:9100" 18:53:31 DEBUG "2300.6666666666665" 18:53:31 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:53:31 DEBUG "value": [ 18:53:31 DEBUG }, 18:53:31 DEBUG ] 18:53:31 DEBUG }, 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG } 18:53:31 DEBUG "value": [ 18:53:31 DEBUG "0" 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG ] 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG ] 18:53:31 DEBUG "6775.966666666666" 18:53:31 DEBUG } 18:53:31 DEBUG "0.0029232090284582453" 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "0.077" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "pod": "ds-idrepo-1" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "pod": "ds-idrepo-0" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "0" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "0" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "430.6743305761952" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG { 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:53:31 DEBUG { 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG "metric": { 18:53:31 DEBUG { 18:53:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG }, 18:53:31 DEBUG "data": { 18:53:31 DEBUG "value": [ 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG 18:53:31 DEBUG "result": [ 18:53:31 DEBUG 18:53:31 DEBUG { 18:53:31 DEBUG 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG "metric": { 18:53:31 DEBUG [...] 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:53:31 DEBUG 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG }, 18:53:31 DEBUG 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:53:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:53:31 DEBUG 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG }, 18:53:31 DEBUG 18:53:31 DEBUG { 18:53:31 DEBUG "value": [ 18:53:31 DEBUG [...] 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG 18:53:31 DEBUG "data": { 18:53:31 DEBUG "0" 18:53:31 DEBUG 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG ] 18:53:31 DEBUG 18:53:31 DEBUG "result": [ 18:53:31 DEBUG } 18:53:31 DEBUG "pod": "idm-b85f9d978-j5pbm" 18:53:31 DEBUG { 18:53:31 DEBUG ] 18:53:31 DEBUG }, 18:53:31 DEBUG "metric": { 18:53:31 DEBUG } 18:53:31 DEBUG "value": [ 18:53:31 DEBUG "pod": "ds-idrepo-1" 18:53:31 DEBUG } 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG }, 18:53:31 DEBUG "37791816.066171326" 18:53:31 DEBUG "value": [ 18:53:31 DEBUG ] 18:53:31 DEBUG 18:53:31 DEBUG } 18:53:31 DEBUG 18:53:31 DEBUG ] 18:53:31 DEBUG 18:53:31 DEBUG } 18:53:31 DEBUG [...] 18:53:31 DEBUG } 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "pod": "ds-idrepo-0" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "0" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "6.694078705678157" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "instance": "10.142.0.107:9100" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "instance": "10.142.0.43:9100" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "5798.3" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "2300.2666666666664" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "pod": "ds-idrepo-1" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "pod": "ds-idrepo-0" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "0" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203611" 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "pod": "ds-idrepo-1" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "pod": "ds-idrepo-0" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "0" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG { 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "2300.7" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG }, 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG "value": [ 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "1617317888" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "instance": "10.142.0.107:9100" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "instance": "10.142.0.43:9100" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "0.000500000000000019" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203611" 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "413342" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 INFO [http_cmd]: http status code OK 18:53:31 DEBUG --- status code --- 18:53:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:53:31 DEBUG --- http response --- 18:53:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:53:31 DEBUG ----- output ----- 18:53:31 DEBUG { 18:53:31 DEBUG "status": "success", 18:53:31 DEBUG "data": { 18:53:31 DEBUG "resultType": "vector", 18:53:31 DEBUG "result": [ 18:53:31 DEBUG { 18:53:31 DEBUG "metric": { 18:53:31 DEBUG "pod": "ds-idrepo-1" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG [...] 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG 18:53:31 DEBUG "pod": "ds-cts-2" 18:53:31 DEBUG }, 18:53:31 DEBUG "value": [ 18:53:31 DEBUG 1713203611, 18:53:31 DEBUG "0" 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG ] 18:53:31 DEBUG } 18:53:31 DEBUG } 18:53:31 INFO 18:53:31 INFO [loop_until]: kubectl --namespace=xlou top pods 18:53:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:53:31 INFO [loop_until]: OK (rc = 0) 18:53:31 DEBUG --- stdout --- 18:53:31 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12336m 8968Mi am-7d597cf7d6-8vr8f 11565m 8997Mi am-7d597cf7d6-9wz9h 15557m 8807Mi ds-cts-0 15657m 8176Mi ds-cts-1 10736m 8378Mi ds-cts-2 10211m 8521Mi ds-idrepo-0 12912m 4904Mi ds-idrepo-1 2018m 4341Mi ds-idrepo-2 1956m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6836m 1454Mi 18:53:31 DEBUG --- stderr --- 18:53:31 DEBUG 18:53:31 INFO 18:53:31 INFO [loop_until]: kubectl --namespace=xlou top node 18:53:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:53:32 INFO [loop_until]: OK (rc = 0) 18:53:32 DEBUG --- stdout --- 18:53:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5118Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10948m 36% 10523Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12930m 43% 11037Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 16952m 56% 3073Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1990m 6% 7114Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16558m 55% 12485Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2983Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12381m 41% 10516Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2012m 6% 6257Mi 5% gke-xlou-cdm-ds-0f601751-j488 10364m 34% 10050Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2447Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17004m 56% 10460Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12987m 43% 7061Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6823m 42% 5317Mi 9% 18:53:32 DEBUG --- stderr --- 18:53:32 DEBUG 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "instance": "10.142.0.107:9100" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:54:31 DEBUG }, 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "instance": "10.142.0.43:9100" 18:54:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:54:31 DEBUG }, 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "13768260608" 18:54:31 DEBUG "2289.7" 18:54:31 DEBUG ] 18:54:31 DEBUG ] 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG ] 18:54:31 DEBUG ] 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 DEBUG --- http response --- 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203671" 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:54:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "ds-idrepo-1" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "ds-cts-2" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "0" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "2297.0333333333333" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "0" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203671" 18:54:31 ERROR [http_cmd]: ERROR 18:54:31 ERROR --- status code --- 18:54:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:54:31 ERROR --- http response --- 18:54:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:54:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203671" 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG "pod": "ds-idrepo-1" 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG }, 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG "value": [ 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:54:31 DEBUG 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG [...] 18:54:31 DEBUG { 18:54:31 DEBUG 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG 18:54:31 DEBUG "data": { 18:54:31 DEBUG 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "pod": "ds-idrepo-0" 18:54:31 DEBUG "result": [ 18:54:31 DEBUG }, 18:54:31 DEBUG { 18:54:31 DEBUG "value": [ 18:54:31 DEBUG "metric": { 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "instance": "10.142.0.107:9100" 18:54:31 DEBUG "0" 18:54:31 DEBUG }, 18:54:31 DEBUG ] 18:54:31 DEBUG "value": [ 18:54:31 DEBUG } 18:54:31 DEBUG 18:54:31 DEBUG ] 18:54:31 DEBUG 18:54:31 DEBUG } 18:54:31 DEBUG 18:54:31 DEBUG } 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "instance": "10.142.0.43:9100" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "5495.066666666667" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "instance": "10.142.0.107:9100" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "instance": "10.142.0.43:9100" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "0" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "0.077" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "ds-idrepo-1" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "ds-idrepo-0" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "0" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "0" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "4753.463407557844" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "ds-idrepo-1" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "ds-idrepo-0" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "0" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203671" 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "2296.4666666666667" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "0" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "6.6687144059661705" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203671" 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "0" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "ds-idrepo-1" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "ds-idrepo-0" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "0" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "instance": "10.142.0.107:9100" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "instance": "10.142.0.43:9100" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "5698.8" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203671" 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "ds-idrepo-1" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "ds-idrepo-0" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "0" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG { 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "2296.9666666666667" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "instance": "10.142.0.107:9100" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "instance": "10.142.0.43:9100" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "0.00030000000000001136" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "1622216704" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203671" 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "412985.5" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:31 INFO [http_cmd]: http status code OK 18:54:31 DEBUG --- status code --- 18:54:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:54:31 DEBUG --- http response --- 18:54:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:54:31 DEBUG ----- output ----- 18:54:31 DEBUG { 18:54:31 DEBUG "status": "success", 18:54:31 DEBUG "data": { 18:54:31 DEBUG "resultType": "vector", 18:54:31 DEBUG "result": [ 18:54:31 DEBUG { 18:54:31 DEBUG "metric": { 18:54:31 DEBUG "pod": "ds-idrepo-1" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG [...] 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG 18:54:31 DEBUG "pod": "ds-cts-2" 18:54:31 DEBUG }, 18:54:31 DEBUG "value": [ 18:54:31 DEBUG 1713203671, 18:54:31 DEBUG "0" 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG ] 18:54:31 DEBUG } 18:54:31 DEBUG } 18:54:32 INFO 18:54:32 INFO [loop_until]: kubectl --namespace=xlou top pods 18:54:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:54:32 INFO [loop_until]: OK (rc = 0) 18:54:32 DEBUG --- stdout --- 18:54:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12557m 8968Mi am-7d597cf7d6-8vr8f 11531m 9004Mi am-7d597cf7d6-9wz9h 15903m 8771Mi ds-cts-0 15445m 8180Mi ds-cts-1 10869m 8354Mi ds-cts-2 10250m 8508Mi ds-idrepo-0 12738m 4907Mi ds-idrepo-1 2011m 4341Mi ds-idrepo-2 1946m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1023Mi idm-b85f9d978-rl76f 4m 1222Mi lodemon-7c47b75958-dgfm9 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6741m 1455Mi 18:54:32 DEBUG --- stderr --- 18:54:32 DEBUG 18:54:32 INFO 18:54:32 INFO [loop_until]: kubectl --namespace=xlou top node 18:54:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:54:32 INFO [loop_until]: OK (rc = 0) 18:54:32 DEBUG --- stdout --- 18:54:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5129Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10891m 36% 10525Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12605m 42% 11038Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17565m 58% 3063Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2018m 6% 7112Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16265m 54% 12508Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2984Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12361m 41% 10520Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2007m 6% 6268Mi 5% gke-xlou-cdm-ds-0f601751-j488 10600m 35% 10035Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2447Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17076m 57% 10430Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12960m 43% 7063Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6838m 43% 5322Mi 9% 18:54:32 DEBUG --- stderr --- 18:54:32 DEBUG 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203731" 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203731" 18:55:31 DEBUG --- status code --- 18:55:31 ERROR [http_cmd]: ERROR 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 ERROR --- status code --- 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG --- status code --- 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG --- http response --- 18:55:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 ERROR --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:55:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG { 18:55:31 DEBUG { 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "data": { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "result": [ 18:55:31 DEBUG "result": [ 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG { 18:55:31 DEBUG { 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "ds-idrepo-1" 18:55:31 DEBUG }, 18:55:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:55:31 DEBUG "instance": "10.142.0.107:9100" 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "ds-cts-1" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG }, 18:55:31 DEBUG }, 18:55:31 DEBUG "instance": "10.142.0.107:9100" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG "value": [ 18:55:31 DEBUG "value": [ 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "ds-cts-2" 18:55:31 DEBUG }, 18:55:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:55:31 DEBUG "instance": "10.142.0.43:9100" 18:55:31 DEBUG 18:55:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG }, 18:55:31 DEBUG }, 18:55:31 DEBUG "instance": "10.142.0.43:9100" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG "value": [ 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "2286.7666666666664" 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "0" 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG ] 18:55:31 DEBUG "2296.1" 18:55:31 DEBUG "13768852480" 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "0" 18:55:31 DEBUG ] 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG ] 18:55:31 DEBUG "0" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 DEBUG { 18:55:31 DEBUG } 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "instance": "10.142.0.107:9100" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "instance": "10.142.0.43:9100" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG "5263.133333333333" 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG ] 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "ds-idrepo-1" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "ds-idrepo-0" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "0" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "ds-idrepo-1" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "ds-idrepo-0" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "0" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "ds-cts-1" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "ds-cts-0" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "31242261.561257116" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "0.077" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "2295.8" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG { 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:55:31 DEBUG "result": [ 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "pod": "ds-idrepo-1" 18:55:31 DEBUG "data": { 18:55:31 DEBUG }, 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "value": [ 18:55:31 DEBUG "result": [ 18:55:31 DEBUG 18:55:31 DEBUG { 18:55:31 DEBUG 18:55:31 DEBUG "metric": { 18:55:31 DEBUG 18:55:31 DEBUG "pod": "ds-cts-1" 18:55:31 DEBUG [...] 18:55:31 DEBUG }, 18:55:31 DEBUG 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "ds-idrepo-0" 18:55:31 DEBUG 18:55:31 DEBUG }, 18:55:31 DEBUG [...] 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG 18:55:31 DEBUG "0" 18:55:31 DEBUG 18:55:31 DEBUG ] 18:55:31 DEBUG "pod": "ds-cts-0" 18:55:31 DEBUG } 18:55:31 DEBUG }, 18:55:31 DEBUG ] 18:55:31 DEBUG "value": [ 18:55:31 DEBUG } 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG } 18:55:31 DEBUG "43769831.084769554" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203731" 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "ds-cts-1" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "0" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "6.7177663927768165" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203731" 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "ds-idrepo-1" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "ds-idrepo-0" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "0" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "0" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "instance": "10.142.0.107:9100" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "instance": "10.142.0.43:9100" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "5927.0666666666675" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203731" 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "instance": "10.142.0.107:9100" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "instance": "10.142.0.43:9100" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "0.00036666666666652263" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203731" 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "ds-idrepo-1" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "ds-idrepo-0" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "0" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "2292.3" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "1624121344" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203731" 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "412749.5" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:31 INFO [http_cmd]: http status code OK 18:55:31 DEBUG --- status code --- 18:55:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:55:31 DEBUG --- http response --- 18:55:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:55:31 DEBUG ----- output ----- 18:55:31 DEBUG { 18:55:31 DEBUG "status": "success", 18:55:31 DEBUG "data": { 18:55:31 DEBUG "resultType": "vector", 18:55:31 DEBUG "result": [ 18:55:31 DEBUG { 18:55:31 DEBUG "metric": { 18:55:31 DEBUG "pod": "ds-idrepo-1" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG [...] 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG 18:55:31 DEBUG "pod": "ds-cts-2" 18:55:31 DEBUG }, 18:55:31 DEBUG "value": [ 18:55:31 DEBUG 1713203731, 18:55:31 DEBUG "0" 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG ] 18:55:31 DEBUG } 18:55:31 DEBUG } 18:55:32 INFO 18:55:32 INFO [loop_until]: kubectl --namespace=xlou top pods 18:55:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:55:32 INFO [loop_until]: OK (rc = 0) 18:55:32 DEBUG --- stdout --- 18:55:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12417m 9000Mi am-7d597cf7d6-8vr8f 11572m 9004Mi am-7d597cf7d6-9wz9h 15647m 8768Mi ds-cts-0 15406m 8181Mi ds-cts-1 11031m 8358Mi ds-cts-2 10780m 8521Mi ds-idrepo-0 12810m 4928Mi ds-idrepo-1 2053m 4341Mi ds-idrepo-2 1938m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1022Mi idm-b85f9d978-rl76f 3m 1223Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6655m 1456Mi 18:55:32 DEBUG --- stderr --- 18:55:32 DEBUG 18:55:32 INFO 18:55:32 INFO [loop_until]: kubectl --namespace=xlou top node 18:55:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:55:32 INFO [loop_until]: OK (rc = 0) 18:55:32 DEBUG --- stdout --- 18:55:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5125Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10638m 35% 10526Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13071m 43% 11059Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17543m 58% 3077Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1977m 6% 7127Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16016m 53% 12485Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 2996Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12390m 41% 10529Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2041m 6% 6269Mi 5% gke-xlou-cdm-ds-0f601751-j488 10230m 34% 10051Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2442Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16579m 55% 10423Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12878m 43% 7072Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6922m 43% 5322Mi 9% 18:55:32 DEBUG --- stderr --- 18:55:32 DEBUG 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203791" 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203791" 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203791" 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG --- status code --- 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG --- http response --- 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "pod": "ds-idrepo-1" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "pod": "ds-cts-2" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "0" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "pod": "end-user-ui-58878d7668-zg4h2" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "0.000022262214223825668" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203791" 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG --- status code --- 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "instance": "10.142.0.107:9100" 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "pod": "ds-idrepo-1" 18:56:31 DEBUG }, 18:56:31 DEBUG "instance": "10.142.0.107:9100" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "instance": "10.142.0.43:9100" 18:56:31 DEBUG 18:56:31 DEBUG "pod": "ds-idrepo-0" 18:56:31 DEBUG }, 18:56:31 DEBUG "instance": "10.142.0.43:9100" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "5083.7" 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "0" 18:56:31 DEBUG ] 18:56:31 DEBUG "13769491456" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "2296.9" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 ERROR [http_cmd]: ERROR 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 ERROR --- status code --- 18:56:31 DEBUG --- status code --- 18:56:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 ERROR --- http response --- 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:56:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:56:31 DEBUG }, 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG "value": [ 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG "2300.3333333333335" 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:56:31 DEBUG ] 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG } 18:56:31 DEBUG { 18:56:31 DEBUG } 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "instance": "10.142.0.107:9100" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "instance": "10.142.0.43:9100" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "0" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "pod": "ds-idrepo-1" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "pod": "ds-idrepo-0" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "0" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "pod": "end-user-ui-58878d7668-zg4h2" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "0" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "0.077" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "2296.633333333333" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "pod": "ds-idrepo-1" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "pod": "ds-idrepo-0" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "0" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "pod": "end-user-ui-58878d7668-zg4h2" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "0" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203791" 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "pod": "end-user-ui-58878d7668-zg4h2" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "0" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203791" 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "6.8794958662546115" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203791" 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "0" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "pod": "ds-idrepo-1" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "pod": "ds-idrepo-0" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "0" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "instance": "10.142.0.107:9100" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "instance": "10.142.0.43:9100" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "5442.433333333333" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203791" 18:56:31 INFO [http_cmd]: http status code OK 18:56:31 DEBUG --- status code --- 18:56:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:31 DEBUG --- http response --- 18:56:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:56:31 DEBUG ----- output ----- 18:56:31 DEBUG { 18:56:31 DEBUG "status": "success", 18:56:31 DEBUG "data": { 18:56:31 DEBUG "resultType": "vector", 18:56:31 DEBUG "result": [ 18:56:31 DEBUG { 18:56:31 DEBUG "metric": { 18:56:31 DEBUG "instance": "10.142.0.107:9100" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG [...] 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG 18:56:31 DEBUG "instance": "10.142.0.43:9100" 18:56:31 DEBUG }, 18:56:31 DEBUG "value": [ 18:56:31 DEBUG 1713203791, 18:56:31 DEBUG "0.0001666666666665151" 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG ] 18:56:31 DEBUG } 18:56:31 DEBUG } 18:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203792" 18:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203792" 18:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203792" 18:56:32 INFO [http_cmd]: http status code OK 18:56:32 DEBUG --- status code --- 18:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:32 DEBUG --- http response --- 18:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:56:32 DEBUG ----- output ----- 18:56:32 DEBUG { 18:56:32 DEBUG "status": "success", 18:56:32 DEBUG "data": { 18:56:32 DEBUG "resultType": "vector", 18:56:32 DEBUG "result": [ 18:56:32 DEBUG { 18:56:32 DEBUG "metric": { 18:56:32 DEBUG "pod": "ds-idrepo-1" 18:56:32 DEBUG }, 18:56:32 DEBUG "value": [ 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG [...] 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG "pod": "ds-idrepo-0" 18:56:32 DEBUG }, 18:56:32 DEBUG "value": [ 18:56:32 DEBUG 1713203792, 18:56:32 DEBUG "0" 18:56:32 DEBUG ] 18:56:32 DEBUG } 18:56:32 DEBUG ] 18:56:32 DEBUG } 18:56:32 DEBUG } 18:56:32 INFO [http_cmd]: http status code OK 18:56:32 DEBUG --- status code --- 18:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:32 DEBUG --- http response --- 18:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:56:32 DEBUG ----- output ----- 18:56:32 DEBUG { 18:56:32 DEBUG "status": "success", 18:56:32 DEBUG "data": { 18:56:32 DEBUG "resultType": "vector", 18:56:32 DEBUG "result": [ 18:56:32 DEBUG { 18:56:32 DEBUG "metric": { 18:56:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:56:32 DEBUG }, 18:56:32 DEBUG "value": [ 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG [...] 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:56:32 DEBUG }, 18:56:32 DEBUG "value": [ 18:56:32 DEBUG 1713203792, 18:56:32 DEBUG "2289.366666666667" 18:56:32 DEBUG ] 18:56:32 DEBUG } 18:56:32 DEBUG ] 18:56:32 DEBUG } 18:56:32 DEBUG } 18:56:32 INFO [http_cmd]: http status code OK 18:56:32 DEBUG --- status code --- 18:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:32 DEBUG --- http response --- 18:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:56:32 DEBUG ----- output ----- 18:56:32 DEBUG { 18:56:32 DEBUG "status": "success", 18:56:32 DEBUG "data": { 18:56:32 DEBUG "resultType": "vector", 18:56:32 DEBUG "result": [ 18:56:32 DEBUG { 18:56:32 DEBUG "metric": { 18:56:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:56:32 DEBUG }, 18:56:32 DEBUG "value": [ 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG [...] 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:56:32 DEBUG }, 18:56:32 DEBUG "value": [ 18:56:32 DEBUG 1713203792, 18:56:32 DEBUG "1626718208" 18:56:32 DEBUG ] 18:56:32 DEBUG } 18:56:32 DEBUG ] 18:56:32 DEBUG } 18:56:32 DEBUG } 18:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203792" 18:56:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203792" 18:56:32 INFO [http_cmd]: http status code OK 18:56:32 DEBUG --- status code --- 18:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:32 DEBUG --- http response --- 18:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:56:32 DEBUG ----- output ----- 18:56:32 DEBUG { 18:56:32 DEBUG "status": "success", 18:56:32 DEBUG "data": { 18:56:32 DEBUG "resultType": "vector", 18:56:32 DEBUG "result": [ 18:56:32 DEBUG { 18:56:32 DEBUG "metric": { 18:56:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:56:32 DEBUG }, 18:56:32 DEBUG "value": [ 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG [...] 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:56:32 DEBUG }, 18:56:32 DEBUG "value": [ 18:56:32 DEBUG 1713203792, 18:56:32 DEBUG "412930.5" 18:56:32 DEBUG ] 18:56:32 DEBUG } 18:56:32 DEBUG ] 18:56:32 DEBUG } 18:56:32 DEBUG } 18:56:32 INFO [http_cmd]: http status code OK 18:56:32 DEBUG --- status code --- 18:56:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:56:32 DEBUG --- http response --- 18:56:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:56:32 DEBUG ----- output ----- 18:56:32 DEBUG { 18:56:32 DEBUG "status": "success", 18:56:32 DEBUG "data": { 18:56:32 DEBUG "resultType": "vector", 18:56:32 DEBUG "result": [ 18:56:32 DEBUG { 18:56:32 DEBUG "metric": { 18:56:32 DEBUG "pod": "ds-idrepo-1" 18:56:32 DEBUG }, 18:56:32 DEBUG "value": [ 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG [...] 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG 18:56:32 DEBUG "pod": "ds-cts-2" 18:56:32 DEBUG }, 18:56:32 DEBUG "value": [ 18:56:32 DEBUG 1713203792, 18:56:32 DEBUG "0" 18:56:32 DEBUG ] 18:56:32 DEBUG } 18:56:32 DEBUG ] 18:56:32 DEBUG } 18:56:32 DEBUG } 18:56:32 INFO 18:56:32 INFO [loop_until]: kubectl --namespace=xlou top pods 18:56:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:56:32 INFO [loop_until]: OK (rc = 0) 18:56:32 DEBUG --- stdout --- 18:56:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12729m 8993Mi am-7d597cf7d6-8vr8f 11866m 9005Mi am-7d597cf7d6-9wz9h 15988m 8775Mi ds-cts-0 15103m 8184Mi ds-cts-1 10780m 8386Mi ds-cts-2 10802m 8518Mi ds-idrepo-0 13065m 4928Mi ds-idrepo-1 2065m 4340Mi ds-idrepo-2 1954m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6826m 1458Mi 18:56:32 DEBUG --- stderr --- 18:56:32 DEBUG 18:56:32 INFO 18:56:32 INFO [loop_until]: kubectl --namespace=xlou top node 18:56:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:56:32 INFO [loop_until]: OK (rc = 0) 18:56:32 DEBUG --- stdout --- 18:56:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5133Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10815m 36% 10541Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13159m 44% 11064Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17526m 58% 3058Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2015m 6% 7119Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16511m 55% 12479Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2992Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12437m 41% 10525Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2063m 6% 6260Mi 5% gke-xlou-cdm-ds-0f601751-j488 10702m 35% 10057Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2441Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17012m 56% 10434Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12944m 43% 7077Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6900m 43% 5318Mi 9% 18:56:32 DEBUG --- stderr --- 18:56:32 DEBUG 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203851" 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203851" 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203851" 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203851" 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203851" 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203851" 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203851" 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203851" 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203851" 18:57:31 ERROR [http_cmd]: ERROR 18:57:31 ERROR --- status code --- 18:57:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:57:31 ERROR --- http response --- 18:57:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:57:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:57:31 INFO [http_cmd]: http status code OK 18:57:31 DEBUG --- status code --- 18:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:31 DEBUG --- http response --- 18:57:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:57:31 DEBUG ----- output ----- 18:57:31 DEBUG { 18:57:31 DEBUG "status": "success", 18:57:31 DEBUG "data": { 18:57:31 DEBUG "resultType": "vector", 18:57:31 DEBUG "result": [ 18:57:31 DEBUG { 18:57:31 DEBUG "metric": { 18:57:31 DEBUG "pod": "ds-idrepo-1" 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 18:57:31 INFO [http_cmd]: http status code OK 18:57:31 DEBUG 18:57:31 INFO [http_cmd]: http status code OK 18:57:31 INFO [http_cmd]: http status code OK 18:57:31 INFO [http_cmd]: http status code OK 18:57:31 DEBUG --- status code --- 18:57:31 DEBUG 18:57:31 DEBUG --- status code --- 18:57:31 INFO [http_cmd]: http status code OK 18:57:31 DEBUG --- status code --- 18:57:31 DEBUG --- status code --- 18:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:31 DEBUG [...] 18:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:31 DEBUG --- status code --- 18:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:31 DEBUG --- http response --- 18:57:31 INFO [http_cmd]: http status code OK 18:57:31 DEBUG 18:57:31 DEBUG --- http response --- 18:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:31 DEBUG --- http response --- 18:57:31 DEBUG --- http response --- 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203851" 18:57:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:57:31 DEBUG --- status code --- 18:57:31 DEBUG 18:57:31 DEBUG --- http response --- 18:57:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:57:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:57:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:57:31 DEBUG ----- output ----- 18:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:31 INFO [http_cmd]: http status code OK 18:57:31 DEBUG 18:57:31 DEBUG ----- output ----- 18:57:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:57:31 DEBUG ----- output ----- 18:57:31 DEBUG ----- output ----- 18:57:31 DEBUG { 18:57:31 DEBUG --- http response --- 18:57:31 DEBUG --- status code --- 18:57:31 DEBUG "pod": "ds-cts-2" 18:57:31 DEBUG { 18:57:31 DEBUG ----- output ----- 18:57:31 DEBUG { 18:57:31 DEBUG { 18:57:31 DEBUG "status": "success", 18:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:57:31 DEBUG }, 18:57:31 DEBUG "status": "success", 18:57:31 DEBUG { 18:57:31 DEBUG "status": "success", 18:57:31 DEBUG "status": "success", 18:57:31 DEBUG "data": { 18:57:31 DEBUG --- http response --- 18:57:31 DEBUG "value": [ 18:57:31 DEBUG "data": { 18:57:31 DEBUG "status": "success", 18:57:31 DEBUG "data": { 18:57:31 DEBUG "data": { 18:57:31 DEBUG "resultType": "vector", 18:57:31 DEBUG 1713203851, 18:57:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:57:31 DEBUG "resultType": "vector", 18:57:31 DEBUG "data": { 18:57:31 DEBUG "resultType": "vector", 18:57:31 DEBUG "resultType": "vector", 18:57:31 DEBUG "result": [ 18:57:31 DEBUG "0.00006666666666666667" 18:57:31 DEBUG ----- output ----- 18:57:31 DEBUG "result": [ 18:57:31 DEBUG "resultType": "vector", 18:57:31 DEBUG "result": [ 18:57:31 DEBUG "result": [ 18:57:31 DEBUG { 18:57:31 DEBUG ] 18:57:31 DEBUG { 18:57:31 DEBUG { 18:57:31 DEBUG "result": [ 18:57:31 DEBUG { 18:57:31 DEBUG { 18:57:31 DEBUG "metric": { 18:57:31 DEBUG } 18:57:31 DEBUG "status": "success", 18:57:31 DEBUG "metric": { 18:57:31 DEBUG { 18:57:31 DEBUG "metric": { 18:57:31 DEBUG "metric": { 18:57:31 DEBUG "pod": "ds-idrepo-1" 18:57:31 DEBUG ] 18:57:31 DEBUG "data": { 18:57:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:57:31 DEBUG "metric": { 18:57:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:57:31 DEBUG "instance": "10.142.0.107:9100" 18:57:31 DEBUG }, 18:57:31 DEBUG } 18:57:31 DEBUG "resultType": "vector", 18:57:31 DEBUG }, 18:57:31 DEBUG "instance": "10.142.0.107:9100" 18:57:31 DEBUG }, 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG } 18:57:31 DEBUG "result": [ 18:57:31 DEBUG "value": [ 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 18:57:31 DEBUG { 18:57:31 DEBUG 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG "metric": { 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG "instance": "10.142.0.107:9100" 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG [...] 18:57:31 DEBUG }, 18:57:31 DEBUG [...] 18:57:31 DEBUG 18:57:31 DEBUG [...] 18:57:31 DEBUG [...] 18:57:31 DEBUG 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 18:57:31 DEBUG [...] 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG "pod": "ds-idrepo-0" 18:57:31 DEBUG 18:57:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:57:31 DEBUG 18:57:31 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 18:57:31 DEBUG "instance": "10.142.0.43:9100" 18:57:31 DEBUG }, 18:57:31 DEBUG [...] 18:57:31 DEBUG }, 18:57:31 DEBUG "instance": "10.142.0.43:9100" 18:57:31 DEBUG }, 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 18:57:31 DEBUG "value": [ 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 1713203851, 18:57:31 DEBUG 18:57:31 DEBUG 1713203851, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 1713203851, 18:57:31 DEBUG 1713203851, 18:57:31 DEBUG "0" 18:57:31 DEBUG 18:57:31 DEBUG "2289.7666666666664" 18:57:31 DEBUG 1713203851, 18:57:31 DEBUG "2.8204193818210106" 18:57:31 DEBUG "5204.3" 18:57:31 DEBUG ] 18:57:31 DEBUG "instance": "10.142.0.43:9100" 18:57:31 DEBUG ] 18:57:31 DEBUG "13770099712" 18:57:31 DEBUG ] 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG }, 18:57:31 DEBUG } 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG } 18:57:31 DEBUG ] 18:57:31 DEBUG "value": [ 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG ] 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG 1713203851, 18:57:31 DEBUG } 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG } 18:57:31 DEBUG } 18:57:31 DEBUG "0" 18:57:31 DEBUG } 18:57:31 DEBUG } 18:57:31 DEBUG } 18:57:31 DEBUG } 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG } 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG } 18:57:31 INFO [http_cmd]: http status code OK 18:57:31 DEBUG --- status code --- 18:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:31 DEBUG --- http response --- 18:57:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:57:31 DEBUG ----- output ----- 18:57:31 DEBUG { 18:57:31 DEBUG "status": "success", 18:57:31 DEBUG "data": { 18:57:31 DEBUG "resultType": "vector", 18:57:31 DEBUG "result": [ 18:57:31 DEBUG { 18:57:31 DEBUG "metric": { 18:57:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG [...] 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 1713203851, 18:57:31 DEBUG "2296.7" 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG } 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203851" 18:57:31 INFO [http_cmd]: http status code OK 18:57:31 DEBUG --- status code --- 18:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:31 DEBUG --- http response --- 18:57:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:57:31 DEBUG ----- output ----- 18:57:31 DEBUG { 18:57:31 DEBUG "status": "success", 18:57:31 DEBUG "data": { 18:57:31 DEBUG "resultType": "vector", 18:57:31 DEBUG "result": [ 18:57:31 DEBUG { 18:57:31 DEBUG "metric": { 18:57:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG [...] 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 1713203851, 18:57:31 DEBUG "0" 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG } 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203851" 18:57:31 INFO [http_cmd]: http status code OK 18:57:31 DEBUG --- status code --- 18:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:31 DEBUG --- http response --- 18:57:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:57:31 DEBUG ----- output ----- 18:57:31 DEBUG { 18:57:31 DEBUG "status": "success", 18:57:31 DEBUG "data": { 18:57:31 DEBUG "resultType": "vector", 18:57:31 DEBUG "result": [ 18:57:31 DEBUG { 18:57:31 DEBUG "metric": { 18:57:31 DEBUG "pod": "ds-idrepo-1" 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG [...] 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG "pod": "ds-idrepo-0" 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 1713203851, 18:57:31 DEBUG "0" 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG } 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203851" 18:57:31 INFO [http_cmd]: http status code OK 18:57:31 DEBUG --- status code --- 18:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:31 DEBUG --- http response --- 18:57:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:57:31 DEBUG ----- output ----- 18:57:31 DEBUG { 18:57:31 DEBUG "status": "success", 18:57:31 DEBUG "data": { 18:57:31 DEBUG "resultType": "vector", 18:57:31 DEBUG "result": [ 18:57:31 DEBUG { 18:57:31 DEBUG "metric": { 18:57:31 DEBUG "pod": "ds-idrepo-1" 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG [...] 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG "pod": "ds-idrepo-0" 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 1713203851, 18:57:31 DEBUG "0" 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG } 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203851" 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203851" 18:57:31 INFO [http_cmd]: http status code OK 18:57:31 DEBUG --- status code --- 18:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:31 DEBUG --- http response --- 18:57:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:57:31 DEBUG ----- output ----- 18:57:31 DEBUG { 18:57:31 DEBUG "status": "success", 18:57:31 DEBUG "data": { 18:57:31 DEBUG "resultType": "vector", 18:57:31 DEBUG "result": [ 18:57:31 DEBUG { 18:57:31 DEBUG "metric": { 18:57:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG [...] 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 1713203851, 18:57:31 DEBUG "0.077" 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG } 18:57:31 INFO [http_cmd]: http status code OK 18:57:31 DEBUG --- status code --- 18:57:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:31 DEBUG --- http response --- 18:57:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:57:31 DEBUG ----- output ----- 18:57:31 DEBUG { 18:57:31 DEBUG "status": "success", 18:57:31 DEBUG "data": { 18:57:31 DEBUG "resultType": "vector", 18:57:31 DEBUG "result": [ 18:57:31 DEBUG { 18:57:31 DEBUG "metric": { 18:57:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG [...] 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG 18:57:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:57:31 DEBUG }, 18:57:31 DEBUG "value": [ 18:57:31 DEBUG 1713203851, 18:57:31 DEBUG "2289.2666666666664" 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG ] 18:57:31 DEBUG } 18:57:31 DEBUG } 18:57:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203851" 18:57:32 INFO [http_cmd]: http status code OK 18:57:32 DEBUG --- status code --- 18:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:32 DEBUG --- http response --- 18:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 18:57:32 DEBUG ----- output ----- 18:57:32 DEBUG { 18:57:32 DEBUG "status": "success", 18:57:32 DEBUG "data": { 18:57:32 DEBUG "resultType": "vector", 18:57:32 DEBUG "result": [ 18:57:32 DEBUG { 18:57:32 DEBUG "metric": { 18:57:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG [...] 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG "pod": "overseer-0-5584f6c4db-bcn5q" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 1713203851, 18:57:32 DEBUG "0" 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG } 18:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203852" 18:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203852" 18:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203852" 18:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203852" 18:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203852" 18:57:32 INFO [http_cmd]: http status code OK 18:57:32 DEBUG --- status code --- 18:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:32 DEBUG --- http response --- 18:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:57:32 DEBUG ----- output ----- 18:57:32 DEBUG { 18:57:32 DEBUG "status": "success", 18:57:32 DEBUG "data": { 18:57:32 DEBUG "resultType": "vector", 18:57:32 DEBUG "result": [ 18:57:32 DEBUG { 18:57:32 DEBUG "metric": { 18:57:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG [...] 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 1713203852, 18:57:32 DEBUG "61243.06942938143" 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG } 18:57:32 INFO [http_cmd]: http status code OK 18:57:32 DEBUG --- status code --- 18:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:32 DEBUG --- http response --- 18:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:57:32 DEBUG ----- output ----- 18:57:32 DEBUG { 18:57:32 DEBUG "status": "success", 18:57:32 DEBUG "data": { 18:57:32 DEBUG "resultType": "vector", 18:57:32 DEBUG "result": [ 18:57:32 DEBUG { 18:57:32 DEBUG "metric": { 18:57:32 DEBUG "pod": "ds-idrepo-1" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG [...] 18:57:32 DEBUG 18:57:32 INFO [http_cmd]: http status code OK 18:57:32 DEBUG 18:57:32 DEBUG --- status code --- 18:57:32 DEBUG 18:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:32 DEBUG "pod": "ds-idrepo-0" 18:57:32 DEBUG --- http response --- 18:57:32 DEBUG }, 18:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:57:32 DEBUG "value": [ 18:57:32 DEBUG ----- output ----- 18:57:32 DEBUG 1713203852, 18:57:32 DEBUG { 18:57:32 DEBUG "0" 18:57:32 DEBUG "status": "success", 18:57:32 DEBUG ] 18:57:32 DEBUG "data": { 18:57:32 DEBUG } 18:57:32 DEBUG "resultType": "vector", 18:57:32 DEBUG ] 18:57:32 DEBUG "result": [ 18:57:32 DEBUG } 18:57:32 DEBUG { 18:57:32 DEBUG } 18:57:32 DEBUG "metric": { 18:57:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG [...] 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 1713203852, 18:57:32 DEBUG "0" 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG } 18:57:32 INFO [http_cmd]: http status code OK 18:57:32 INFO [http_cmd]: http status code OK 18:57:32 DEBUG --- status code --- 18:57:32 DEBUG --- status code --- 18:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:32 DEBUG --- http response --- 18:57:32 DEBUG --- http response --- 18:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:57:32 DEBUG ----- output ----- 18:57:32 DEBUG { 18:57:32 DEBUG "status": "success", 18:57:32 DEBUG "data": { 18:57:32 DEBUG "resultType": "vector", 18:57:32 DEBUG "result": [ 18:57:32 DEBUG { 18:57:32 DEBUG "metric": { 18:57:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG [...] 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 1713203852, 18:57:32 DEBUG "6.794542170048614" 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG } 18:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:57:32 DEBUG ----- output ----- 18:57:32 DEBUG { 18:57:32 DEBUG "status": "success", 18:57:32 DEBUG "data": { 18:57:32 DEBUG "resultType": "vector", 18:57:32 DEBUG "result": [ 18:57:32 DEBUG { 18:57:32 DEBUG "metric": { 18:57:32 DEBUG "instance": "10.142.0.107:9100" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG [...] 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG "instance": "10.142.0.43:9100" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 1713203852, 18:57:32 DEBUG "5609.233333333334" 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG } 18:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203852" 18:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203852" 18:57:32 INFO [http_cmd]: http status code OK 18:57:32 DEBUG --- status code --- 18:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:32 DEBUG --- http response --- 18:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:57:32 DEBUG ----- output ----- 18:57:32 DEBUG { 18:57:32 DEBUG "status": "success", 18:57:32 DEBUG "data": { 18:57:32 DEBUG "resultType": "vector", 18:57:32 DEBUG "result": [ 18:57:32 DEBUG { 18:57:32 DEBUG "metric": { 18:57:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG [...] 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 1713203852, 18:57:32 DEBUG "1626664960" 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG } 18:57:32 INFO [http_cmd]: http status code OK 18:57:32 DEBUG --- status code --- 18:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:32 DEBUG --- http response --- 18:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:57:32 DEBUG ----- output ----- 18:57:32 DEBUG { 18:57:32 DEBUG "status": "success", 18:57:32 DEBUG "data": { 18:57:32 DEBUG "resultType": "vector", 18:57:32 DEBUG "result": [ 18:57:32 DEBUG { 18:57:32 DEBUG "metric": { 18:57:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG [...] 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 1713203852, 18:57:32 DEBUG "2290.1" 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG } 18:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203852" 18:57:32 INFO [http_cmd]: http status code OK 18:57:32 DEBUG --- status code --- 18:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:32 DEBUG --- http response --- 18:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:57:32 DEBUG ----- output ----- 18:57:32 DEBUG { 18:57:32 DEBUG "status": "success", 18:57:32 DEBUG "data": { 18:57:32 DEBUG "resultType": "vector", 18:57:32 DEBUG "result": [ 18:57:32 DEBUG { 18:57:32 DEBUG "metric": { 18:57:32 DEBUG "instance": "10.142.0.107:9100" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG [...] 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG "instance": "10.142.0.43:9100" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 1713203852, 18:57:32 DEBUG "0.00033333333333350387" 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG } 18:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203852" 18:57:32 INFO [http_cmd]: http status code OK 18:57:32 DEBUG --- status code --- 18:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:32 DEBUG --- http response --- 18:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:57:32 DEBUG ----- output ----- 18:57:32 DEBUG { 18:57:32 DEBUG "status": "success", 18:57:32 DEBUG "data": { 18:57:32 DEBUG "resultType": "vector", 18:57:32 DEBUG "result": [ 18:57:32 DEBUG { 18:57:32 DEBUG "metric": { 18:57:32 DEBUG "pod": "ds-idrepo-1" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG [...] 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG "pod": "ds-idrepo-0" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 1713203852, 18:57:32 DEBUG "0" 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG } 18:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203852" 18:57:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203852" 18:57:32 INFO [http_cmd]: http status code OK 18:57:32 DEBUG --- status code --- 18:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:32 DEBUG --- http response --- 18:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:57:32 DEBUG ----- output ----- 18:57:32 DEBUG { 18:57:32 DEBUG "status": "success", 18:57:32 DEBUG "data": { 18:57:32 DEBUG "resultType": "vector", 18:57:32 DEBUG "result": [ 18:57:32 DEBUG { 18:57:32 DEBUG "metric": { 18:57:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG [...] 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 1713203852, 18:57:32 DEBUG "413374" 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG } 18:57:32 INFO [http_cmd]: http status code OK 18:57:32 DEBUG --- status code --- 18:57:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:57:32 DEBUG --- http response --- 18:57:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:57:32 DEBUG ----- output ----- 18:57:32 DEBUG { 18:57:32 DEBUG "status": "success", 18:57:32 DEBUG "data": { 18:57:32 DEBUG "resultType": "vector", 18:57:32 DEBUG "result": [ 18:57:32 DEBUG { 18:57:32 DEBUG "metric": { 18:57:32 DEBUG "pod": "ds-idrepo-1" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG [...] 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG 18:57:32 DEBUG "pod": "ds-cts-2" 18:57:32 DEBUG }, 18:57:32 DEBUG "value": [ 18:57:32 DEBUG 1713203852, 18:57:32 DEBUG "0" 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG ] 18:57:32 DEBUG } 18:57:32 DEBUG } 18:57:32 INFO 18:57:32 INFO [loop_until]: kubectl --namespace=xlou top pods 18:57:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:57:32 INFO [loop_until]: OK (rc = 0) 18:57:32 DEBUG --- stdout --- 18:57:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12640m 8978Mi am-7d597cf7d6-8vr8f 11808m 9003Mi am-7d597cf7d6-9wz9h 15811m 8769Mi ds-cts-0 15279m 8188Mi ds-cts-1 10874m 8367Mi ds-cts-2 10694m 8522Mi ds-idrepo-0 13045m 4927Mi ds-idrepo-1 1986m 4340Mi ds-idrepo-2 1923m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 4m 1222Mi lodemon-7c47b75958-dgfm9 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6767m 1460Mi 18:57:32 DEBUG --- stderr --- 18:57:32 DEBUG 18:57:32 INFO 18:57:32 INFO [loop_until]: kubectl --namespace=xlou top node 18:57:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:57:32 INFO [loop_until]: OK (rc = 0) 18:57:32 DEBUG --- stdout --- 18:57:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5138Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10876m 36% 10543Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13017m 43% 11045Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 16986m 56% 3055Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2024m 6% 7123Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16785m 56% 12502Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2981Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12363m 41% 10517Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2038m 6% 6262Mi 5% gke-xlou-cdm-ds-0f601751-j488 10631m 35% 10065Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2448Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17117m 57% 10435Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13038m 43% 7064Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6898m 43% 5320Mi 9% 18:57:32 DEBUG --- stderr --- 18:57:32 DEBUG 18:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203911" 18:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203911" 18:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203911" 18:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203911" 18:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203911" 18:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203911" 18:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203911" 18:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203911" 18:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203911" 18:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203911" 18:58:31 ERROR [http_cmd]: ERROR 18:58:31 INFO [http_cmd]: http status code OK 18:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203911" 18:58:31 INFO [http_cmd]: http status code OK 18:58:31 ERROR --- status code --- 18:58:31 DEBUG --- status code --- 18:58:31 DEBUG --- status code --- 18:58:31 INFO [http_cmd]: http status code OK 18:58:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:58:31 INFO [http_cmd]: http status code OK 18:58:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:31 DEBUG --- status code --- 18:58:31 ERROR --- http response --- 18:58:31 DEBUG --- status code --- 18:58:31 DEBUG --- http response --- 18:58:31 DEBUG --- http response --- 18:58:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:58:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:58:31 DEBUG --- http response --- 18:58:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:58:31 DEBUG --- http response --- 18:58:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:58:31 DEBUG ----- output ----- 18:58:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:58:31 DEBUG ----- output ----- 18:58:31 INFO [http_cmd]: http status code OK 18:58:31 INFO [http_cmd]: http status code OK 18:58:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:58:31 INFO [http_cmd]: http status code OK 18:58:31 DEBUG { 18:58:31 DEBUG ----- output ----- 18:58:31 DEBUG { 18:58:31 DEBUG --- status code --- 18:58:31 DEBUG --- status code --- 18:58:31 DEBUG ----- output ----- 18:58:31 DEBUG --- status code --- 18:58:31 DEBUG "status": "success", 18:58:31 DEBUG { 18:58:31 DEBUG "status": "success", 18:58:31 INFO [http_cmd]: http status code OK 18:58:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:31 DEBUG { 18:58:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:31 DEBUG "data": { 18:58:31 DEBUG "status": "success", 18:58:31 DEBUG "data": { 18:58:31 DEBUG --- status code --- 18:58:31 DEBUG --- http response --- 18:58:31 DEBUG --- http response --- 18:58:31 DEBUG "status": "success", 18:58:31 DEBUG --- http response --- 18:58:31 DEBUG "resultType": "vector", 18:58:31 DEBUG "data": { 18:58:31 DEBUG "resultType": "vector", 18:58:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:58:31 DEBUG "data": { 18:58:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:58:31 DEBUG "result": [ 18:58:31 DEBUG "resultType": "vector", 18:58:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:58:31 DEBUG "result": [ 18:58:31 DEBUG --- http response --- 18:58:31 DEBUG ----- output ----- 18:58:31 DEBUG "resultType": "vector", 18:58:31 DEBUG ----- output ----- 18:58:31 DEBUG { 18:58:31 DEBUG "result": [ 18:58:31 DEBUG { 18:58:31 INFO [http_cmd]: http status code OK 18:58:31 DEBUG --- status code --- 18:58:31 DEBUG { 18:58:31 DEBUG "result": [ 18:58:31 DEBUG { 18:58:31 DEBUG "metric": { 18:58:31 DEBUG { 18:58:31 DEBUG "metric": { 18:58:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:58:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:31 DEBUG "status": "success", 18:58:31 DEBUG { 18:58:31 DEBUG "status": "success", 18:58:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:58:31 DEBUG "metric": { 18:58:31 DEBUG "instance": "10.142.0.107:9100" 18:58:31 DEBUG ----- output ----- 18:58:31 DEBUG --- http response --- 18:58:31 DEBUG "data": { 18:58:31 DEBUG "metric": { 18:58:31 DEBUG "data": { 18:58:31 DEBUG }, 18:58:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:58:31 DEBUG }, 18:58:31 DEBUG { 18:58:31 DEBUG "resultType": "vector", 18:58:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:58:31 DEBUG "pod": "ds-cts-1" 18:58:31 INFO [http_cmd]: http status code OK 18:58:31 DEBUG "resultType": "vector", 18:58:31 DEBUG "value": [ 18:58:31 DEBUG }, 18:58:31 DEBUG "value": [ 18:58:31 DEBUG "status": "success", 18:58:31 DEBUG "result": [ 18:58:31 DEBUG ----- output ----- 18:58:31 DEBUG }, 18:58:31 DEBUG --- status code --- 18:58:31 DEBUG "result": [ 18:58:31 DEBUG 18:58:31 DEBUG "value": [ 18:58:31 DEBUG 18:58:31 DEBUG "data": { 18:58:31 DEBUG { 18:58:31 DEBUG { 18:58:31 DEBUG "value": [ 18:58:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:31 DEBUG { 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG "resultType": "vector", 18:58:31 DEBUG "metric": { 18:58:31 DEBUG "status": "success", 18:58:31 DEBUG 18:58:31 DEBUG --- http response --- 18:58:31 DEBUG "metric": { 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG "result": [ 18:58:31 DEBUG "pod": "ds-idrepo-1" 18:58:31 DEBUG "data": { 18:58:31 DEBUG 18:58:31 DEBUG "pod": "ds-idrepo-1" 18:58:31 DEBUG [...] 18:58:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:58:31 DEBUG 18:58:31 DEBUG [...] 18:58:31 DEBUG { 18:58:31 DEBUG }, 18:58:31 DEBUG "resultType": "vector", 18:58:31 DEBUG 18:58:31 DEBUG }, 18:58:31 DEBUG 18:58:31 DEBUG ----- output ----- 18:58:31 DEBUG [...] 18:58:31 DEBUG 18:58:31 DEBUG "metric": { 18:58:31 DEBUG "value": [ 18:58:31 DEBUG "result": [ 18:58:31 DEBUG [...] 18:58:31 DEBUG "value": [ 18:58:31 DEBUG 18:58:31 DEBUG { 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG "pod": "ds-cts-1" 18:58:31 DEBUG 18:58:31 DEBUG { 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG "status": "success", 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG }, 18:58:31 DEBUG 18:58:31 DEBUG "metric": { 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG "data": { 18:58:31 DEBUG 18:58:31 DEBUG "instance": "10.142.0.43:9100" 18:58:31 DEBUG "value": [ 18:58:31 DEBUG 18:58:31 DEBUG "instance": "10.142.0.107:9100" 18:58:31 DEBUG 18:58:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:58:31 DEBUG [...] 18:58:31 DEBUG "resultType": "vector", 18:58:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:58:31 DEBUG }, 18:58:31 DEBUG 18:58:31 DEBUG [...] 18:58:31 DEBUG }, 18:58:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:58:31 DEBUG }, 18:58:31 DEBUG 18:58:31 DEBUG "result": [ 18:58:31 DEBUG }, 18:58:31 DEBUG "value": [ 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG "value": [ 18:58:31 DEBUG }, 18:58:31 DEBUG "value": [ 18:58:31 DEBUG 18:58:31 DEBUG { 18:58:31 DEBUG "value": [ 18:58:31 DEBUG 1713203911, 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG "value": [ 18:58:31 DEBUG 1713203911, 18:58:31 DEBUG 18:58:31 DEBUG "metric": { 18:58:31 DEBUG 1713203911, 18:58:31 DEBUG "13770738688" 18:58:31 DEBUG [...] 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG 1713203911, 18:58:31 DEBUG "2294.1" 18:58:31 DEBUG "pod": "ds-cts-2" 18:58:31 DEBUG "instance": "10.142.0.107:9100" 18:58:31 DEBUG "2301.9" 18:58:31 DEBUG ] 18:58:31 DEBUG 18:58:31 DEBUG "pod": "ds-idrepo-0" 18:58:31 DEBUG 18:58:31 DEBUG "0" 18:58:31 DEBUG ] 18:58:31 DEBUG }, 18:58:31 DEBUG }, 18:58:31 DEBUG ] 18:58:31 DEBUG } 18:58:31 DEBUG 18:58:31 DEBUG }, 18:58:31 DEBUG [...] 18:58:31 DEBUG ] 18:58:31 DEBUG } 18:58:31 DEBUG "value": [ 18:58:31 DEBUG "value": [ 18:58:31 DEBUG } 18:58:31 DEBUG ] 18:58:31 DEBUG 18:58:31 DEBUG "value": [ 18:58:31 DEBUG 18:58:31 DEBUG } 18:58:31 DEBUG ] 18:58:31 DEBUG 1713203911, 18:58:31 DEBUG 18:58:31 DEBUG ] 18:58:31 DEBUG } 18:58:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:58:31 DEBUG 1713203911, 18:58:31 DEBUG 18:58:31 DEBUG ] 18:58:31 DEBUG } 18:58:31 DEBUG "0" 18:58:31 DEBUG 18:58:31 DEBUG } 18:58:31 DEBUG } 18:58:31 DEBUG }, 18:58:31 DEBUG "0" 18:58:31 DEBUG 18:58:31 DEBUG } 18:58:31 DEBUG } 18:58:31 DEBUG ] 18:58:31 DEBUG 18:58:31 DEBUG } 18:58:31 DEBUG "value": [ 18:58:31 DEBUG ] 18:58:31 DEBUG "instance": "10.142.0.43:9100" 18:58:31 DEBUG } 18:58:31 DEBUG } 18:58:31 DEBUG [...] 18:58:31 DEBUG 1713203911, 18:58:31 DEBUG } 18:58:31 DEBUG }, 18:58:31 DEBUG ] 18:58:31 DEBUG 18:58:31 DEBUG "0.00003054110877501327" 18:58:31 DEBUG ] 18:58:31 DEBUG "value": [ 18:58:31 DEBUG } 18:58:31 DEBUG 18:58:31 DEBUG ] 18:58:31 DEBUG } 18:58:31 DEBUG 1713203911, 18:58:31 DEBUG } 18:58:31 DEBUG 18:58:31 DEBUG } 18:58:31 DEBUG } 18:58:31 DEBUG "0" 18:58:31 DEBUG "instance": "10.142.0.43:9100" 18:58:31 DEBUG ] 18:58:31 DEBUG ] 18:58:31 DEBUG }, 18:58:31 DEBUG } 18:58:31 DEBUG } 18:58:31 DEBUG "value": [ 18:58:31 DEBUG } 18:58:31 DEBUG ] 18:58:31 DEBUG 1713203911, 18:58:31 DEBUG } 18:58:31 DEBUG "6727.766666666666" 18:58:31 DEBUG } 18:58:31 DEBUG ] 18:58:31 DEBUG } 18:58:31 DEBUG ] 18:58:31 DEBUG } 18:58:31 DEBUG } 18:58:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203911" 18:58:31 INFO [http_cmd]: http status code OK 18:58:31 DEBUG --- status code --- 18:58:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:31 DEBUG --- http response --- 18:58:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:58:31 DEBUG ----- output ----- 18:58:31 DEBUG { 18:58:31 DEBUG "status": "success", 18:58:31 DEBUG "data": { 18:58:31 DEBUG "resultType": "vector", 18:58:31 DEBUG "result": [ 18:58:31 DEBUG { 18:58:31 DEBUG "metric": { 18:58:31 DEBUG "pod": "ds-idrepo-1" 18:58:31 DEBUG }, 18:58:31 DEBUG "value": [ 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG [...] 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG 18:58:31 DEBUG "pod": "ds-idrepo-0" 18:58:31 DEBUG }, 18:58:31 DEBUG "value": [ 18:58:31 DEBUG 1713203911, 18:58:31 DEBUG "0" 18:58:31 DEBUG ] 18:58:31 DEBUG } 18:58:31 DEBUG ] 18:58:31 DEBUG } 18:58:31 DEBUG } 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203912" 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "2300.8333333333335" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203912" 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "pod": "ds-idrepo-1" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "pod": "ds-idrepo-0" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "0" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203912" 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203912" 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203912" 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203912" 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "0.077" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "pod": "ds-idrepo-1" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "pod": "ds-idrepo-0" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "0" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "1628164096" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203912" 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "instance": "10.142.0.107:9100" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "instance": "10.142.0.43:9100" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "5707.033333333334" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "pod": "ds-cts-1" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "9101814.093056945" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203912" 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203912" 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203912" 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "pod": "ds-cts-1" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "317145511.26133895" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "0" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203912" 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "6.683973551766718" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "pod": "ds-idrepo-1" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "pod": "ds-idrepo-0" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "0" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203912" 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203912" 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "2303.9" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "instance": "10.142.0.107:9100" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "instance": "10.142.0.43:9100" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "0.00033333333333350387" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203912" 18:58:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203912" 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "413157.5" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 INFO [http_cmd]: http status code OK 18:58:32 DEBUG --- status code --- 18:58:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:58:32 DEBUG --- http response --- 18:58:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:58:32 DEBUG ----- output ----- 18:58:32 DEBUG { 18:58:32 DEBUG "status": "success", 18:58:32 DEBUG "data": { 18:58:32 DEBUG "resultType": "vector", 18:58:32 DEBUG "result": [ 18:58:32 DEBUG { 18:58:32 DEBUG "metric": { 18:58:32 DEBUG "pod": "ds-idrepo-1" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG [...] 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG 18:58:32 DEBUG "pod": "ds-cts-2" 18:58:32 DEBUG }, 18:58:32 DEBUG "value": [ 18:58:32 DEBUG 1713203912, 18:58:32 DEBUG "0" 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG ] 18:58:32 DEBUG } 18:58:32 DEBUG } 18:58:32 INFO 18:58:32 INFO [loop_until]: kubectl --namespace=xlou top pods 18:58:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:58:32 INFO [loop_until]: OK (rc = 0) 18:58:32 DEBUG --- stdout --- 18:58:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12610m 8975Mi am-7d597cf7d6-8vr8f 11414m 9003Mi am-7d597cf7d6-9wz9h 15848m 8768Mi ds-cts-0 15319m 8189Mi ds-cts-1 10939m 8370Mi ds-cts-2 10179m 8543Mi ds-idrepo-0 12918m 4927Mi ds-idrepo-1 1963m 4340Mi ds-idrepo-2 1854m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1022Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6808m 1461Mi 18:58:32 DEBUG --- stderr --- 18:58:32 DEBUG 18:58:32 INFO 18:58:32 INFO [loop_until]: kubectl --namespace=xlou top node 18:58:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:58:32 INFO [loop_until]: OK (rc = 0) 18:58:32 DEBUG --- stdout --- 18:58:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5122Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10876m 36% 10544Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12963m 43% 11045Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17537m 58% 3065Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1899m 6% 7124Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16453m 55% 12508Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 67m 0% 2982Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 11989m 40% 10529Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1986m 6% 6262Mi 5% gke-xlou-cdm-ds-0f601751-j488 10419m 34% 10055Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2454Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17073m 57% 10433Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12988m 43% 7071Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6828m 42% 5320Mi 9% 18:58:32 DEBUG --- stderr --- 18:58:32 DEBUG 18:59:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203971" 18:59:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203971" 18:59:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203971" 18:59:31 INFO [http_cmd]: http status code OK 18:59:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713203971" 18:59:31 DEBUG --- status code --- 18:59:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:31 DEBUG --- http response --- 18:59:31 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 18:59:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203971" 18:59:31 ERROR [http_cmd]: ERROR 18:59:31 ERROR --- status code --- 18:59:31 ERROR http status code is 400 (expected [200, 401, 403, 404]) 18:59:31 ERROR --- http response --- 18:59:31 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 18:59:31 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 18:59:31 INFO [http_cmd]: http status code OK 18:59:31 DEBUG --- status code --- 18:59:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:31 DEBUG --- http response --- 18:59:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203971" 18:59:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 18:59:31 DEBUG ----- output ----- 18:59:31 DEBUG { 18:59:31 DEBUG "status": "success", 18:59:31 DEBUG "data": { 18:59:31 DEBUG "resultType": "vector", 18:59:31 DEBUG "result": [ 18:59:31 DEBUG { 18:59:31 DEBUG "metric": { 18:59:31 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:59:31 DEBUG }, 18:59:31 DEBUG "value": [ 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG [...] 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG "pod": "am-7d597cf7d6-8vr8f" 18:59:31 DEBUG }, 18:59:31 DEBUG "value": [ 18:59:31 DEBUG 1713203971, 18:59:31 DEBUG "11.677366019944733" 18:59:31 DEBUG ] 18:59:31 DEBUG } 18:59:31 DEBUG ] 18:59:31 DEBUG } 18:59:31 DEBUG } 18:59:31 INFO [http_cmd]: http status code OK 18:59:31 INFO [http_cmd]: http status code OK 18:59:31 DEBUG --- status code --- 18:59:31 DEBUG --- status code --- 18:59:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:31 DEBUG --- http response --- 18:59:31 DEBUG --- http response --- 18:59:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:59:31 DEBUG ----- output ----- 18:59:31 DEBUG { 18:59:31 DEBUG "status": "success", 18:59:31 DEBUG "data": { 18:59:31 DEBUG "resultType": "vector", 18:59:31 DEBUG "result": [ 18:59:31 DEBUG { 18:59:31 DEBUG "metric": { 18:59:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:59:31 DEBUG }, 18:59:31 DEBUG "value": [ 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG [...] 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:59:31 DEBUG }, 18:59:31 DEBUG "value": [ 18:59:31 DEBUG 1713203971, 18:59:31 DEBUG "2294.866666666667" 18:59:31 DEBUG ] 18:59:31 DEBUG } 18:59:31 DEBUG ] 18:59:31 DEBUG } 18:59:31 DEBUG } 18:59:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:59:31 DEBUG ----- output ----- 18:59:31 DEBUG { 18:59:31 DEBUG "status": "success", 18:59:31 DEBUG "data": { 18:59:31 DEBUG "resultType": "vector", 18:59:31 DEBUG "result": [ 18:59:31 DEBUG { 18:59:31 DEBUG "metric": { 18:59:31 DEBUG "instance": "10.142.0.107:9100" 18:59:31 DEBUG }, 18:59:31 DEBUG "value": [ 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG [...] 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG "instance": "10.142.0.43:9100" 18:59:31 DEBUG }, 18:59:31 DEBUG "value": [ 18:59:31 DEBUG 1713203971, 18:59:31 DEBUG "5449.733333333333" 18:59:31 DEBUG ] 18:59:31 DEBUG } 18:59:31 DEBUG ] 18:59:31 DEBUG } 18:59:31 DEBUG } 18:59:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203971" 18:59:31 INFO [http_cmd]: http status code OK 18:59:31 DEBUG --- status code --- 18:59:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:31 DEBUG --- http response --- 18:59:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:59:31 DEBUG ----- output ----- 18:59:31 DEBUG { 18:59:31 DEBUG "status": "success", 18:59:31 DEBUG "data": { 18:59:31 DEBUG "resultType": "vector", 18:59:31 DEBUG "result": [ 18:59:31 DEBUG { 18:59:31 DEBUG "metric": { 18:59:31 DEBUG "instance": "10.142.0.107:9100" 18:59:31 DEBUG }, 18:59:31 DEBUG "value": [ 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG [...] 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG "instance": "10.142.0.43:9100" 18:59:31 DEBUG }, 18:59:31 DEBUG "value": [ 18:59:31 DEBUG 1713203971, 18:59:31 DEBUG "0" 18:59:31 DEBUG ] 18:59:31 DEBUG } 18:59:31 DEBUG ] 18:59:31 DEBUG } 18:59:31 DEBUG } 18:59:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203971" 18:59:31 INFO [http_cmd]: http status code OK 18:59:31 DEBUG --- status code --- 18:59:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:31 DEBUG --- http response --- 18:59:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:59:31 DEBUG ----- output ----- 18:59:31 DEBUG { 18:59:31 DEBUG "status": "success", 18:59:31 DEBUG "data": { 18:59:31 DEBUG "resultType": "vector", 18:59:31 DEBUG "result": [ 18:59:31 DEBUG { 18:59:31 DEBUG "metric": { 18:59:31 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:59:31 DEBUG }, 18:59:31 DEBUG "value": [ 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG [...] 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:59:31 DEBUG }, 18:59:31 DEBUG "value": [ 18:59:31 DEBUG 1713203971, 18:59:31 DEBUG "2310.2666666666664" 18:59:31 DEBUG ] 18:59:31 DEBUG } 18:59:31 DEBUG ] 18:59:31 DEBUG } 18:59:31 DEBUG } 18:59:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203971" 18:59:31 INFO [http_cmd]: http status code OK 18:59:31 DEBUG --- status code --- 18:59:31 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:31 DEBUG --- http response --- 18:59:31 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 18:59:31 DEBUG ----- output ----- 18:59:31 DEBUG { 18:59:31 DEBUG "status": "success", 18:59:31 DEBUG "data": { 18:59:31 DEBUG "resultType": "vector", 18:59:31 DEBUG "result": [ 18:59:31 DEBUG { 18:59:31 DEBUG "metric": { 18:59:31 DEBUG "pod": "ds-idrepo-1" 18:59:31 DEBUG }, 18:59:31 DEBUG "value": [ 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG [...] 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG 18:59:31 DEBUG "pod": "ds-cts-2" 18:59:31 DEBUG }, 18:59:31 DEBUG "value": [ 18:59:31 DEBUG 1713203971, 18:59:31 DEBUG "0.000052002222222222226" 18:59:31 DEBUG ] 18:59:31 DEBUG } 18:59:31 DEBUG ] 18:59:31 DEBUG } 18:59:31 DEBUG } 18:59:31 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203971" 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203972" 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203972" 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "pod": "ds-idrepo-1" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "pod": "ds-idrepo-0" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203971, 18:59:32 DEBUG "0" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "instance": "10.142.0.107:9100" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "instance": "10.142.0.43:9100" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203971, 18:59:32 DEBUG "13771377664" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "pod": "am-7d597cf7d6-8vr8f" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "0" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "pod": "ds-idrepo-1" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "pod": "ds-idrepo-0" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "0" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203972" 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203972" 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "2309.6" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "pod": "ds-idrepo-1" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "pod": "ds-idrepo-0" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "0" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713203972" 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203972" 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713203972" 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203972" 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "pod": "ds-idrepo-1" 18:59:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:59:32 DEBUG }, 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:59:32 DEBUG [...] 18:59:32 DEBUG [...] 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG { 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "data": { 18:59:32 DEBUG "pod": "ds-idrepo-0" 18:59:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG }, 18:59:32 DEBUG }, 18:59:32 DEBUG "result": [ 18:59:32 DEBUG "value": [ 18:59:32 DEBUG "value": [ 18:59:32 DEBUG { 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "0" 18:59:32 DEBUG "0.077" 18:59:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:59:32 DEBUG ] 18:59:32 DEBUG ] 18:59:32 DEBUG }, 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713203972" 18:59:32 DEBUG "value": [ 18:59:32 DEBUG ] 18:59:32 DEBUG ] 18:59:32 DEBUG 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 DEBUG 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 DEBUG 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203972" 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "1630150656" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "instance": "10.142.0.107:9100" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "instance": "10.142.0.43:9100" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "5527.133333333334" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "pod": "am-7d597cf7d6-8vr8f" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "7598135.702629193" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "6.706992013627788" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203972" 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203972" 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "pod": "am-7d597cf7d6-8vr8f" 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG }, 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG "value": [ 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG "121208.92112420671" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:59:32 DEBUG } 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "0" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203972" 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "pod": "ds-idrepo-1" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "pod": "ds-idrepo-0" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "0" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203972" 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713203972" 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "2310.3333333333335" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "instance": "10.142.0.107:9100" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "instance": "10.142.0.43:9100" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "0.00023333333333350008" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713203972" 18:59:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713203972" 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "413094.5" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO [http_cmd]: http status code OK 18:59:32 DEBUG --- status code --- 18:59:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 18:59:32 DEBUG --- http response --- 18:59:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 18:59:32 DEBUG ----- output ----- 18:59:32 DEBUG { 18:59:32 DEBUG "status": "success", 18:59:32 DEBUG "data": { 18:59:32 DEBUG "resultType": "vector", 18:59:32 DEBUG "result": [ 18:59:32 DEBUG { 18:59:32 DEBUG "metric": { 18:59:32 DEBUG "pod": "ds-idrepo-1" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG [...] 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG 18:59:32 DEBUG "pod": "ds-cts-2" 18:59:32 DEBUG }, 18:59:32 DEBUG "value": [ 18:59:32 DEBUG 1713203972, 18:59:32 DEBUG "0" 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG ] 18:59:32 DEBUG } 18:59:32 DEBUG } 18:59:32 INFO 18:59:32 INFO [loop_until]: kubectl --namespace=xlou top pods 18:59:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:59:32 INFO [loop_until]: OK (rc = 0) 18:59:32 DEBUG --- stdout --- 18:59:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 13092m 8974Mi am-7d597cf7d6-8vr8f 11813m 9005Mi am-7d597cf7d6-9wz9h 15372m 8770Mi ds-cts-0 15699m 8194Mi ds-cts-1 11271m 8372Mi ds-cts-2 10212m 8535Mi ds-idrepo-0 13148m 4928Mi ds-idrepo-1 1954m 4340Mi ds-idrepo-2 1954m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6716m 1464Mi 18:59:32 DEBUG --- stderr --- 18:59:32 DEBUG 18:59:32 INFO 18:59:32 INFO [loop_until]: kubectl --namespace=xlou top node 18:59:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 18:59:32 INFO [loop_until]: OK (rc = 0) 18:59:32 DEBUG --- stdout --- 18:59:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5129Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11009m 36% 10544Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13081m 43% 11047Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17670m 59% 3076Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1991m 6% 7125Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16797m 56% 12519Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2990Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12554m 42% 10525Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2014m 6% 6258Mi 5% gke-xlou-cdm-ds-0f601751-j488 10625m 35% 10057Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2445Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17314m 57% 10487Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13023m 43% 7071Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6641m 41% 5319Mi 9% 18:59:32 DEBUG --- stderr --- 18:59:32 DEBUG 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204032" 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 ERROR [http_cmd]: ERROR 19:00:32 DEBUG --- status code --- 19:00:32 ERROR --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 ERROR --- http response --- 19:00:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:00:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:00:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "2289.4" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG --- http response --- 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG { 19:00:32 DEBUG { 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:00:32 DEBUG "data": { 19:00:32 DEBUG "data": { 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG { 19:00:32 DEBUG "result": [ 19:00:32 DEBUG "result": [ 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG { 19:00:32 DEBUG { 19:00:32 DEBUG "data": { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:00:32 DEBUG "instance": "10.142.0.107:9100" 19:00:32 DEBUG "result": [ 19:00:32 DEBUG }, 19:00:32 DEBUG }, 19:00:32 DEBUG { 19:00:32 DEBUG "value": [ 19:00:32 DEBUG "value": [ 19:00:32 DEBUG "metric": { 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "instance": "10.142.0.107:9100" 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG }, 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "value": [ 19:00:32 DEBUG [...] 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG "pod": "ds-cts-2" 19:00:32 DEBUG "instance": "10.142.0.43:9100" 19:00:32 DEBUG 19:00:32 DEBUG }, 19:00:32 DEBUG }, 19:00:32 DEBUG 19:00:32 DEBUG "value": [ 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "instance": "10.142.0.43:9100" 19:00:32 DEBUG "10.744533268765114" 19:00:32 DEBUG "5770.433333333333" 19:00:32 DEBUG }, 19:00:32 DEBUG ] 19:00:32 DEBUG ] 19:00:32 DEBUG "value": [ 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG ] 19:00:32 DEBUG ] 19:00:32 DEBUG "0.00033333333333333365" 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "ds-idrepo-1" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "pod": "ds-idrepo-0" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "0" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "ds-idrepo-1" 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:00:32 DEBUG "pod": "ds-idrepo-1" 19:00:32 DEBUG }, 19:00:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:00:32 DEBUG }, 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "pod": "ds-idrepo-0" 19:00:32 DEBUG 19:00:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:00:32 DEBUG "pod": "ds-cts-2" 19:00:32 DEBUG }, 19:00:32 DEBUG "pod": "ds-cts-2" 19:00:32 DEBUG }, 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "0" 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "2302.733333333333" 19:00:32 DEBUG "0.000019224444444444444" 19:00:32 DEBUG ] 19:00:32 DEBUG "0" 19:00:32 DEBUG ] 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "instance": "10.142.0.107:9100" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "instance": "10.142.0.43:9100" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "13772319744" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "0.077" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "2303.7" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "ds-idrepo-1" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "pod": "ds-idrepo-0" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "0" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204032" 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "pod": "ds-cts-2" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "53652641.1622276" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "instance": "10.142.0.107:9100" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "instance": "10.142.0.43:9100" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "6047.5" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "ds-idrepo-1" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "pod": "ds-idrepo-0" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "0" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204032" 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "1632960512" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "6.67643588184255" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG "data": { 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG "result": [ 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "pod": "ds-cts-2" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "288876499.37046003" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "0" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "ds-idrepo-1" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "pod": "ds-idrepo-0" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "0" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204032" 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "2293.3" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "instance": "10.142.0.107:9100" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "instance": "10.142.0.43:9100" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "0.0005333333333330377" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204032" 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "413457.5" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO [http_cmd]: http status code OK 19:00:32 DEBUG --- status code --- 19:00:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:00:32 DEBUG --- http response --- 19:00:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:00:32 DEBUG ----- output ----- 19:00:32 DEBUG { 19:00:32 DEBUG "status": "success", 19:00:32 DEBUG "data": { 19:00:32 DEBUG "resultType": "vector", 19:00:32 DEBUG "result": [ 19:00:32 DEBUG { 19:00:32 DEBUG "metric": { 19:00:32 DEBUG "pod": "ds-idrepo-1" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG [...] 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG 19:00:32 DEBUG "pod": "ds-cts-2" 19:00:32 DEBUG }, 19:00:32 DEBUG "value": [ 19:00:32 DEBUG 1713204032, 19:00:32 DEBUG "0" 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG ] 19:00:32 DEBUG } 19:00:32 DEBUG } 19:00:32 INFO 19:00:32 INFO [loop_until]: kubectl --namespace=xlou top pods 19:00:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:00:32 INFO [loop_until]: OK (rc = 0) 19:00:32 DEBUG --- stdout --- 19:00:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12727m 8995Mi am-7d597cf7d6-8vr8f 11639m 9005Mi am-7d597cf7d6-9wz9h 16243m 8871Mi ds-cts-0 15318m 8196Mi ds-cts-1 10983m 8379Mi ds-cts-2 10312m 8536Mi ds-idrepo-0 12914m 4928Mi ds-idrepo-1 1990m 4341Mi ds-idrepo-2 1890m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1022Mi idm-b85f9d978-rl76f 4m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6654m 1464Mi 19:00:32 DEBUG --- stderr --- 19:00:32 DEBUG 19:00:32 INFO 19:00:32 INFO [loop_until]: kubectl --namespace=xlou top node 19:00:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:00:32 INFO [loop_until]: OK (rc = 0) 19:00:32 DEBUG --- stdout --- 19:00:32 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5094Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10986m 36% 10549Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13058m 43% 11059Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17558m 58% 3072Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2019m 6% 7121Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16734m 56% 12467Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2995Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12572m 42% 10526Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1996m 6% 6268Mi 5% gke-xlou-cdm-ds-0f601751-j488 10715m 35% 10074Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2449Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17159m 57% 10529Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12983m 43% 7082Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6826m 42% 5321Mi 9% 19:00:32 DEBUG --- stderr --- 19:00:32 DEBUG 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 ERROR [http_cmd]: ERROR 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 ERROR --- status code --- 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG --- status code --- 19:01:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 ERROR --- http response --- 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:01:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:01:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "pod": "ds-cts-1" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "7.177141313514343" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "instance": "10.142.0.107:9100" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "instance": "10.142.0.43:9100" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "8525.866666666667" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG { 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:01:32 DEBUG "data": { 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "instance": "10.142.0.107:9100" 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG }, 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "value": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "pod": "ds-idrepo-1" 19:01:32 DEBUG 19:01:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:01:32 DEBUG }, 19:01:32 DEBUG 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG [...] 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG [...] 19:01:32 DEBUG "instance": "10.142.0.43:9100" 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG [...] 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG 19:01:32 DEBUG }, 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG 19:01:32 DEBUG "value": [ 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:01:32 DEBUG 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:01:32 DEBUG "pod": "ds-idrepo-0" 19:01:32 DEBUG "0" 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG }, 19:01:32 DEBUG ] 19:01:32 DEBUG { 19:01:32 DEBUG }, 19:01:32 DEBUG { 19:01:32 DEBUG { 19:01:32 DEBUG { 19:01:32 DEBUG "value": [ 19:01:32 DEBUG } 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "value": [ 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG ] 19:01:32 DEBUG "data": { 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "2302.866666666667" 19:01:32 DEBUG "data": { 19:01:32 DEBUG "data": { 19:01:32 DEBUG "0" 19:01:32 DEBUG } 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "data": { 19:01:32 DEBUG ] 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG "result": [ 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG } 19:01:32 DEBUG "result": [ 19:01:32 DEBUG "result": [ 19:01:32 DEBUG } 19:01:32 DEBUG { 19:01:32 DEBUG "result": [ 19:01:32 DEBUG ] 19:01:32 DEBUG { 19:01:32 DEBUG { 19:01:32 DEBUG ] 19:01:32 DEBUG "metric": { 19:01:32 DEBUG { 19:01:32 DEBUG } 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "metric": { 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG } 19:01:32 DEBUG "pod": "ds-idrepo-1" 19:01:32 DEBUG "metric": { 19:01:32 DEBUG } 19:01:32 DEBUG "pod": "ds-cts-1" 19:01:32 DEBUG "instance": "10.142.0.107:9100" 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG } 19:01:32 DEBUG }, 19:01:32 DEBUG "pod": "ds-idrepo-1" 19:01:32 DEBUG }, 19:01:32 DEBUG }, 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG "value": [ 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG "value": [ 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG { 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG [...] 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "data": { 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "result": [ 19:01:32 DEBUG "pod": "ds-idrepo-0" 19:01:32 DEBUG 19:01:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:01:32 DEBUG "instance": "10.142.0.43:9100" 19:01:32 DEBUG { 19:01:32 DEBUG }, 19:01:32 DEBUG "pod": "ds-cts-2" 19:01:32 DEBUG }, 19:01:32 DEBUG }, 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "value": [ 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG "value": [ 19:01:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG }, 19:01:32 DEBUG "0" 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "0" 19:01:32 DEBUG "13773446144" 19:01:32 DEBUG "value": [ 19:01:32 DEBUG ] 19:01:32 DEBUG "0.000033333333333333335" 19:01:32 DEBUG ] 19:01:32 DEBUG ] 19:01:32 DEBUG 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 DEBUG 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG ] 19:01:32 DEBUG 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 DEBUG [...] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 DEBUG 19:01:32 DEBUG } 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "2294.9" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "2294.4333333333334" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:01:32 DEBUG } 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "0.077" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "pod": "ds-idrepo-1" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "pod": "ds-idrepo-0" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "0" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "pod": "ds-cts-1" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "6192466.057103113" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204092" 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "instance": "10.142.0.107:9100" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "instance": "10.142.0.43:9100" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "5172.766666666667" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "pod": "ds-idrepo-1" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "pod": "ds-idrepo-0" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "0" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204092" 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "1633058816" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204092" 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "6.652512252527052" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "pod": "ds-cts-1" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "275940136.231751" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "instance": "10.142.0.107:9100" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "instance": "10.142.0.43:9100" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "0.00020000000000000757" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "0" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "pod": "ds-idrepo-1" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "pod": "ds-idrepo-0" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "0" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "2295.5333333333333" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204092" 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "413789.5" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO [http_cmd]: http status code OK 19:01:32 DEBUG --- status code --- 19:01:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:01:32 DEBUG --- http response --- 19:01:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:01:32 DEBUG ----- output ----- 19:01:32 DEBUG { 19:01:32 DEBUG "status": "success", 19:01:32 DEBUG "data": { 19:01:32 DEBUG "resultType": "vector", 19:01:32 DEBUG "result": [ 19:01:32 DEBUG { 19:01:32 DEBUG "metric": { 19:01:32 DEBUG "pod": "ds-idrepo-1" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG [...] 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG 19:01:32 DEBUG "pod": "ds-cts-2" 19:01:32 DEBUG }, 19:01:32 DEBUG "value": [ 19:01:32 DEBUG 1713204092, 19:01:32 DEBUG "0" 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG ] 19:01:32 DEBUG } 19:01:32 DEBUG } 19:01:32 INFO 19:01:32 INFO [loop_until]: kubectl --namespace=xlou top pods 19:01:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:01:32 INFO [loop_until]: OK (rc = 0) 19:01:32 DEBUG --- stdout --- 19:01:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12847m 8981Mi am-7d597cf7d6-8vr8f 11231m 9004Mi am-7d597cf7d6-9wz9h 15803m 8833Mi ds-cts-0 15182m 8216Mi ds-cts-1 10617m 8384Mi ds-cts-2 10546m 8534Mi ds-idrepo-0 13077m 4928Mi ds-idrepo-1 1977m 4341Mi ds-idrepo-2 1906m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1022Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6543m 1466Mi 19:01:32 DEBUG --- stderr --- 19:01:32 DEBUG 19:01:32 INFO 19:01:32 INFO [loop_until]: kubectl --namespace=xlou top node 19:01:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:01:33 INFO [loop_until]: OK (rc = 0) 19:01:33 DEBUG --- stdout --- 19:01:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5099Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10830m 36% 10559Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12958m 43% 11051Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 16936m 56% 3060Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2007m 6% 7128Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16510m 55% 12492Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 59m 0% 2975Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12399m 41% 10525Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2046m 6% 6261Mi 5% gke-xlou-cdm-ds-0f601751-j488 10443m 34% 10073Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2444Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17079m 57% 10492Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13039m 43% 7080Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6840m 43% 5322Mi 9% 19:01:33 DEBUG --- stderr --- 19:01:33 DEBUG 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204152" 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204152" 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 ERROR [http_cmd]: ERROR 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 ERROR --- status code --- 19:02:32 DEBUG --- status code --- 19:02:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 ERROR --- http response --- 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:02:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "instance": "10.142.0.107:9100" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "instance": "10.142.0.43:9100" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "5154.333333333333" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "2309.5666666666666" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "0" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "0" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204152" 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG { 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:02:32 DEBUG "data": { 19:02:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG { 19:02:32 DEBUG "result": [ 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG { 19:02:32 DEBUG "data": { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "pod": "ds-idrepo-1" 19:02:32 DEBUG "result": [ 19:02:32 DEBUG }, 19:02:32 DEBUG { 19:02:32 DEBUG "value": [ 19:02:32 DEBUG "metric": { 19:02:32 DEBUG 19:02:32 DEBUG "pod": "ds-idrepo-1" 19:02:32 DEBUG 19:02:32 DEBUG }, 19:02:32 DEBUG 19:02:32 DEBUG "value": [ 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG "pod": "ds-idrepo-0" 19:02:32 DEBUG 19:02:32 DEBUG }, 19:02:32 DEBUG 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "pod": "ds-cts-2" 19:02:32 DEBUG "0" 19:02:32 DEBUG }, 19:02:32 DEBUG ] 19:02:32 DEBUG "value": [ 19:02:32 DEBUG } 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG ] 19:02:32 DEBUG "0.00013333333333333334" 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204152" 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "instance": "10.142.0.107:9100" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "instance": "10.142.0.43:9100" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "0" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "ds-idrepo-1" 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "instance": "10.142.0.107:9100" 19:02:32 DEBUG }, 19:02:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "pod": "ds-idrepo-0" 19:02:32 DEBUG 19:02:32 DEBUG "instance": "10.142.0.43:9100" 19:02:32 DEBUG }, 19:02:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "0" 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "13774089216" 19:02:32 DEBUG ] 19:02:32 DEBUG "2287.0333333333333" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "0.077" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "2286.366666666667" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "ds-idrepo-1" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "pod": "ds-idrepo-0" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "0" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204152" 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204152" 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "instance": "10.142.0.107:9100" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "instance": "10.142.0.43:9100" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "5160.900000000001" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "6.692500358381843" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204152" 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "0" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204152" 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "ds-idrepo-1" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "pod": "ds-idrepo-0" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "0" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "instance": "10.142.0.107:9100" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "instance": "10.142.0.43:9100" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "0.0002666666666669926" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "0" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "1634770944" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "ds-idrepo-1" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "pod": "ds-idrepo-0" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "0" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "0" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "2287.9333333333334" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204152" 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "413450" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO [http_cmd]: http status code OK 19:02:32 DEBUG --- status code --- 19:02:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:02:32 DEBUG --- http response --- 19:02:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:02:32 DEBUG ----- output ----- 19:02:32 DEBUG { 19:02:32 DEBUG "status": "success", 19:02:32 DEBUG "data": { 19:02:32 DEBUG "resultType": "vector", 19:02:32 DEBUG "result": [ 19:02:32 DEBUG { 19:02:32 DEBUG "metric": { 19:02:32 DEBUG "pod": "ds-idrepo-1" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG [...] 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG 19:02:32 DEBUG "pod": "ds-cts-2" 19:02:32 DEBUG }, 19:02:32 DEBUG "value": [ 19:02:32 DEBUG 1713204152, 19:02:32 DEBUG "0" 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG ] 19:02:32 DEBUG } 19:02:32 DEBUG } 19:02:32 INFO 19:02:32 INFO [loop_until]: kubectl --namespace=xlou top pods 19:02:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:02:32 INFO [loop_until]: OK (rc = 0) 19:02:32 DEBUG --- stdout --- 19:02:32 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12640m 8983Mi am-7d597cf7d6-8vr8f 11460m 9006Mi am-7d597cf7d6-9wz9h 16434m 8828Mi ds-cts-0 14864m 8215Mi ds-cts-1 10716m 8387Mi ds-cts-2 10645m 8538Mi ds-idrepo-0 12886m 4928Mi ds-idrepo-1 1967m 4340Mi ds-idrepo-2 1925m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1022Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6654m 1467Mi 19:02:32 DEBUG --- stderr --- 19:02:32 DEBUG 19:02:33 INFO 19:02:33 INFO [loop_until]: kubectl --namespace=xlou top node 19:02:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:02:33 INFO [loop_until]: OK (rc = 0) 19:02:33 DEBUG --- stdout --- 19:02:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5091Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10494m 35% 10566Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12599m 42% 11055Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17496m 58% 3069Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1976m 6% 7113Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16292m 54% 12494Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2986Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12404m 41% 10542Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2042m 6% 6263Mi 5% gke-xlou-cdm-ds-0f601751-j488 10668m 35% 10075Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2452Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17033m 57% 10492Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12979m 43% 7077Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6802m 42% 5319Mi 9% 19:02:33 DEBUG --- stderr --- 19:02:33 DEBUG 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204212" 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204212" 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- status code --- 19:03:32 ERROR [http_cmd]: ERROR 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 ERROR --- status code --- 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:03:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 ERROR --- http response --- 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG { 19:03:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "status": "success", 19:03:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "data": { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG "result": [ 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG { 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "ds-idrepo-1" 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:03:32 DEBUG "instance": "10.142.0.107:9100" 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:03:32 DEBUG }, 19:03:32 DEBUG "pod": "ds-idrepo-1" 19:03:32 DEBUG }, 19:03:32 DEBUG }, 19:03:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG "value": [ 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "pod": "ds-cts-2" 19:03:32 DEBUG 19:03:32 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:03:32 DEBUG "instance": "10.142.0.43:9100" 19:03:32 DEBUG 19:03:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:03:32 DEBUG }, 19:03:32 DEBUG "pod": "ds-idrepo-0" 19:03:32 DEBUG }, 19:03:32 DEBUG }, 19:03:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG "value": [ 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "0" 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "0" 19:03:32 DEBUG "6756.633333333333" 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "2301.0666666666666" 19:03:32 DEBUG ] 19:03:32 DEBUG "0" 19:03:32 DEBUG ] 19:03:32 DEBUG ] 19:03:32 DEBUG "2300.5333333333333" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG { 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "ds-idrepo-1" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "pod": "ds-idrepo-0" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "0" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "0.0029165543025265156" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204212" 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204212" 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "instance": "10.142.0.107:9100" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "instance": "10.142.0.43:9100" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "0.0003333333333333262" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "instance": "10.142.0.107:9100" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "instance": "10.142.0.43:9100" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "13774785536" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "0.0775" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "2299.2" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204212" 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204212" 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "6.679425667650843" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "instance": "10.142.0.107:9100" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "instance": "10.142.0.43:9100" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "5868.5666666666675" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204212" 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "ds-idrepo-1" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "pod": "ds-idrepo-0" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "0" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "ds-idrepo-1" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "pod": "ds-idrepo-0" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "0" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "instance": "10.142.0.107:9100" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "instance": "10.142.0.43:9100" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "0.00036666666666652263" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "502.0838440794313" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204212" 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "ds-idrepo-1" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "pod": "ds-idrepo-0" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "0" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "0" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "1637986304" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "33034857.56312822" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "2300.9666666666667" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "ds-idrepo-1" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "pod": "ds-cts-2" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "0" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204212" 19:03:32 INFO [http_cmd]: http status code OK 19:03:32 DEBUG --- status code --- 19:03:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:03:32 DEBUG --- http response --- 19:03:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:03:32 DEBUG ----- output ----- 19:03:32 DEBUG { 19:03:32 DEBUG "status": "success", 19:03:32 DEBUG "data": { 19:03:32 DEBUG "resultType": "vector", 19:03:32 DEBUG "result": [ 19:03:32 DEBUG { 19:03:32 DEBUG "metric": { 19:03:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG [...] 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG 19:03:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:03:32 DEBUG }, 19:03:32 DEBUG "value": [ 19:03:32 DEBUG 1713204212, 19:03:32 DEBUG "413179.5" 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG ] 19:03:32 DEBUG } 19:03:32 DEBUG } 19:03:32 INFO 19:03:32 INFO [loop_until]: kubectl --namespace=xlou top pods 19:03:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:03:33 INFO [loop_until]: OK (rc = 0) 19:03:33 DEBUG --- stdout --- 19:03:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12663m 8984Mi am-7d597cf7d6-8vr8f 12007m 9009Mi am-7d597cf7d6-9wz9h 15906m 8824Mi ds-cts-0 15497m 8205Mi ds-cts-1 11281m 8388Mi ds-cts-2 10375m 8565Mi ds-idrepo-0 13214m 4928Mi ds-idrepo-1 1954m 4341Mi ds-idrepo-2 1934m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6674m 1469Mi 19:03:33 DEBUG --- stderr --- 19:03:33 DEBUG 19:03:33 INFO 19:03:33 INFO [loop_until]: kubectl --namespace=xlou top node 19:03:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:03:33 INFO [loop_until]: OK (rc = 0) 19:03:33 DEBUG --- stdout --- 19:03:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5097Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10933m 36% 10573Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13072m 43% 11049Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17579m 58% 3074Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1928m 6% 7104Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16610m 55% 12494Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2976Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12506m 41% 10518Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2064m 6% 6264Mi 5% gke-xlou-cdm-ds-0f601751-j488 10388m 34% 10086Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2453Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16959m 56% 10491Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13049m 43% 7091Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6816m 42% 5321Mi 9% 19:03:33 DEBUG --- stderr --- 19:03:33 DEBUG 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "2291.733333333333" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "2294.733333333333" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204272" 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "ds-idrepo-1" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "ds-cts-2" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "0.000033333333333333335" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "ds-idrepo-1" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "ds-idrepo-0" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "0" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 19:04:32 DEBUG --- status code --- 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG { 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204272" 19:04:32 DEBUG "data": { 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG { 19:04:32 DEBUG "result": [ 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG { 19:04:32 DEBUG "data": { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:04:32 DEBUG "result": [ 19:04:32 DEBUG }, 19:04:32 DEBUG { 19:04:32 DEBUG "value": [ 19:04:32 DEBUG "metric": { 19:04:32 DEBUG 19:04:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:04:32 DEBUG 19:04:32 DEBUG }, 19:04:32 DEBUG 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 DEBUG "value": [ 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 ERROR [http_cmd]: ERROR 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 ERROR --- status code --- 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:04:32 DEBUG [...] 19:04:32 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:04:32 ERROR --- http response --- 19:04:32 DEBUG 19:04:32 DEBUG }, 19:04:32 DEBUG 19:04:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:04:32 DEBUG "0.0024777122592910326" 19:04:32 DEBUG }, 19:04:32 DEBUG ] 19:04:32 DEBUG "value": [ 19:04:32 DEBUG } 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG ] 19:04:32 DEBUG "0" 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "instance": "10.142.0.107:9100" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "instance": "10.142.0.43:9100" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "5184.433333333333" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "ds-idrepo-1" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "ds-idrepo-0" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "0" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204272" 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204272" 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "instance": "10.142.0.107:9100" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "instance": "10.142.0.43:9100" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "0" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "instance": "10.142.0.107:9100" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "instance": "10.142.0.43:9100" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "13775453184" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "0.078" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "2290.6666666666665" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204272" 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204272" 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "6.6976469080819125" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "instance": "10.142.0.107:9100" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "instance": "10.142.0.43:9100" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "5615.333333333334" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204272" 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "1636532224" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204272" 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "ds-idrepo-1" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "ds-idrepo-0" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "0" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "instance": "10.142.0.107:9100" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "instance": "10.142.0.43:9100" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "0.00023333333333350008" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "ds-idrepo-1" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "ds-idrepo-0" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "0" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "675.6010061440765" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "ds-idrepo-1" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "ds-idrepo-0" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "0" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "0" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "44450492.59824336" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "ds-idrepo-1" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "ds-cts-2" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "0" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "2291.733333333333" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204272" 19:04:32 INFO [http_cmd]: http status code OK 19:04:32 DEBUG --- status code --- 19:04:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:04:32 DEBUG --- http response --- 19:04:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:04:32 DEBUG ----- output ----- 19:04:32 DEBUG { 19:04:32 DEBUG "status": "success", 19:04:32 DEBUG "data": { 19:04:32 DEBUG "resultType": "vector", 19:04:32 DEBUG "result": [ 19:04:32 DEBUG { 19:04:32 DEBUG "metric": { 19:04:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG [...] 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG 19:04:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:04:32 DEBUG }, 19:04:32 DEBUG "value": [ 19:04:32 DEBUG 1713204272, 19:04:32 DEBUG "412933.5" 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG ] 19:04:32 DEBUG } 19:04:32 DEBUG } 19:04:33 INFO 19:04:33 INFO [loop_until]: kubectl --namespace=xlou top pods 19:04:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:04:33 INFO [loop_until]: OK (rc = 0) 19:04:33 DEBUG --- stdout --- 19:04:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 13188m 8980Mi am-7d597cf7d6-8vr8f 11874m 9008Mi am-7d597cf7d6-9wz9h 15954m 8822Mi ds-cts-0 15108m 8227Mi ds-cts-1 10711m 8390Mi ds-cts-2 10522m 8545Mi ds-idrepo-0 12803m 4929Mi ds-idrepo-1 2029m 4341Mi ds-idrepo-2 1916m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 5m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6639m 1471Mi 19:04:33 DEBUG --- stderr --- 19:04:33 DEBUG 19:04:33 INFO 19:04:33 INFO [loop_until]: kubectl --namespace=xlou top node 19:04:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:04:33 INFO [loop_until]: OK (rc = 0) 19:04:33 DEBUG --- stdout --- 19:04:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5108Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10988m 36% 10567Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13023m 43% 11049Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17587m 58% 3067Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1989m 6% 7124Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16429m 55% 12497Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2970Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12535m 41% 10536Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2043m 6% 6271Mi 5% gke-xlou-cdm-ds-0f601751-j488 10693m 35% 10088Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2445Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17049m 57% 10489Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12980m 43% 7070Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6809m 42% 5320Mi 9% 19:04:33 DEBUG --- stderr --- 19:04:33 DEBUG 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204332" 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "pod": "ds-idrepo-1" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "ds-idrepo-0" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "0" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 ERROR [http_cmd]: ERROR 19:05:32 ERROR --- status code --- 19:05:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:05:32 ERROR --- http response --- 19:05:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:05:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "0.005139520629697764" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204332" 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204332" 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG { 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG { 19:05:32 DEBUG "data": { 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG "data": { 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "data": { 19:05:32 DEBUG { 19:05:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "metric": { 19:05:32 DEBUG }, 19:05:32 DEBUG "result": [ 19:05:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:05:32 DEBUG "value": [ 19:05:32 DEBUG { 19:05:32 DEBUG }, 19:05:32 DEBUG 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG }, 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG }, 19:05:32 DEBUG 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG }, 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG "value": [ 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG "2295.8" 19:05:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG ] 19:05:32 DEBUG }, 19:05:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:05:32 DEBUG "0" 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:05:32 DEBUG } 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:05:32 DEBUG "value": [ 19:05:32 DEBUG ] 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG ] 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG } 19:05:32 DEBUG { 19:05:32 DEBUG } 19:05:32 DEBUG { 19:05:32 DEBUG "2287.5" 19:05:32 DEBUG ] 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204332" 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG "data": { 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG "data": { 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 DEBUG "result": [ 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG "result": [ 19:05:32 DEBUG } 19:05:32 DEBUG { 19:05:32 DEBUG { 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG "pod": "ds-idrepo-1" 19:05:32 DEBUG "instance": "10.142.0.107:9100" 19:05:32 DEBUG { 19:05:32 DEBUG }, 19:05:32 DEBUG }, 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "value": [ 19:05:32 DEBUG "value": [ 19:05:32 DEBUG "data": { 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "result": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG { 19:05:32 DEBUG [...] 19:05:32 DEBUG [...] 19:05:32 DEBUG "metric": { 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "instance": "10.142.0.107:9100" 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG }, 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "value": [ 19:05:32 DEBUG "pod": "ds-cts-2" 19:05:32 DEBUG "instance": "10.142.0.43:9100" 19:05:32 DEBUG 19:05:32 DEBUG }, 19:05:32 DEBUG }, 19:05:32 DEBUG 19:05:32 DEBUG "value": [ 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG [...] 19:05:32 DEBUG "0.000052557777777777776" 19:05:32 DEBUG "13775954944" 19:05:32 DEBUG 19:05:32 DEBUG ] 19:05:32 DEBUG ] 19:05:32 DEBUG 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 DEBUG 19:05:32 DEBUG ] 19:05:32 DEBUG ] 19:05:32 DEBUG "instance": "10.142.0.43:9100" 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 DEBUG }, 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "0" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "instance": "10.142.0.107:9100" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "instance": "10.142.0.43:9100" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "5085.566666666667" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "pod": "ds-idrepo-1" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "ds-idrepo-0" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "0" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "2294.633333333333" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "0.078" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204332" 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204332" 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "6.659475261395265" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "instance": "10.142.0.107:9100" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "instance": "10.142.0.43:9100" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "5437.200000000001" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204332" 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "1639514112" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204332" 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "pod": "ds-idrepo-1" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "ds-idrepo-0" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "0" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "pod": "ds-idrepo-1" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "ds-idrepo-0" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "0" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "instance": "10.142.0.107:9100" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "instance": "10.142.0.43:9100" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "0.00033333333333350387" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "3963.998838672215" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "pod": "ds-idrepo-1" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "ds-idrepo-0" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "0" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "0" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "2157076.034710797" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "pod": "ds-idrepo-1" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "ds-cts-2" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "0" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "2293.5333333333333" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204332" 19:05:32 INFO [http_cmd]: http status code OK 19:05:32 DEBUG --- status code --- 19:05:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:05:32 DEBUG --- http response --- 19:05:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:05:32 DEBUG ----- output ----- 19:05:32 DEBUG { 19:05:32 DEBUG "status": "success", 19:05:32 DEBUG "data": { 19:05:32 DEBUG "resultType": "vector", 19:05:32 DEBUG "result": [ 19:05:32 DEBUG { 19:05:32 DEBUG "metric": { 19:05:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG [...] 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG 19:05:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:05:32 DEBUG }, 19:05:32 DEBUG "value": [ 19:05:32 DEBUG 1713204332, 19:05:32 DEBUG "413114.5" 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG ] 19:05:32 DEBUG } 19:05:32 DEBUG } 19:05:33 INFO 19:05:33 INFO [loop_until]: kubectl --namespace=xlou top pods 19:05:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:05:33 INFO [loop_until]: OK (rc = 0) 19:05:33 DEBUG --- stdout --- 19:05:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12755m 8980Mi am-7d597cf7d6-8vr8f 11578m 9008Mi am-7d597cf7d6-9wz9h 15554m 8823Mi ds-cts-0 15311m 8207Mi ds-cts-1 11130m 8392Mi ds-cts-2 10436m 8561Mi ds-idrepo-0 13105m 4928Mi ds-idrepo-1 1995m 4340Mi ds-idrepo-2 1956m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 4m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6616m 1472Mi 19:05:33 DEBUG --- stderr --- 19:05:33 DEBUG 19:05:33 INFO 19:05:33 INFO [loop_until]: kubectl --namespace=xlou top node 19:05:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:05:33 INFO [loop_until]: OK (rc = 0) 19:05:33 DEBUG --- stdout --- 19:05:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 5108Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10959m 36% 10592Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13049m 43% 11051Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 16906m 56% 3068Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1972m 6% 7133Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16403m 54% 12516Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2964Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12376m 41% 10527Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2029m 6% 6268Mi 5% gke-xlou-cdm-ds-0f601751-j488 10587m 35% 10095Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2445Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17199m 57% 10489Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13003m 43% 7085Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6794m 42% 5322Mi 9% 19:05:33 DEBUG --- stderr --- 19:05:33 DEBUG 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204392" 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "ds-idrepo-1" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "ds-idrepo-0" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "0" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204392" 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 ERROR [http_cmd]: ERROR 19:06:32 ERROR --- status code --- 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204392" 19:06:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG --- status code --- 19:06:32 ERROR --- http response --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:06:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (107 lines): 19:06:32 DEBUG { 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG { 19:06:32 DEBUG "data": { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "result": [ 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG { 19:06:32 DEBUG "result": [ 19:06:32 DEBUG "metric": { 19:06:32 DEBUG { 19:06:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:06:32 DEBUG "metric": { 19:06:32 DEBUG }, 19:06:32 DEBUG "pod": "ds-cts-1" 19:06:32 DEBUG "value": [ 19:06:32 DEBUG }, 19:06:32 DEBUG 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:06:32 DEBUG 19:06:32 DEBUG }, 19:06:32 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:06:32 DEBUG "value": [ 19:06:32 DEBUG }, 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG "2299.7" 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG ] 19:06:32 DEBUG "0" 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "ds-idrepo-1" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "ds-cts-2" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "0.000052557777777777776" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "instance": "10.142.0.107:9100" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "instance": "10.142.0.43:9100" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "6605.4" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "instance": "10.142.0.107:9100" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "instance": "10.142.0.43:9100" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "0.00033333333333333365" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204392" 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "ds-cts-1" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "0.008855574732481884" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "instance": "10.142.0.107:9100" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "instance": "10.142.0.43:9100" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "13776624640" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "2299.6666666666665" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "ds-idrepo-1" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "ds-idrepo-0" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "0" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "2299.6" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "0.078" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204392" 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "6.707595202215631" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204392" 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "instance": "10.142.0.107:9100" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "instance": "10.142.0.43:9100" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "5600.900000000001" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204392" 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204392" 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "instance": "10.142.0.107:9100" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "instance": "10.142.0.43:9100" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "0.00036666666666652263" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "0" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "1640640512" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "ds-idrepo-1" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "ds-idrepo-0" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "0" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "ds-idrepo-1" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "ds-idrepo-0" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "0" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "ds-cts-1" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "0" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "ds-idrepo-1" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "ds-idrepo-0" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "0" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "ds-idrepo-1" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "ds-cts-2" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "0" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "ds-cts-1" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "1555.2640662754575" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "2299.5333333333333" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204392" 19:06:32 INFO [http_cmd]: http status code OK 19:06:32 DEBUG --- status code --- 19:06:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:06:32 DEBUG --- http response --- 19:06:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:06:32 DEBUG ----- output ----- 19:06:32 DEBUG { 19:06:32 DEBUG "status": "success", 19:06:32 DEBUG "data": { 19:06:32 DEBUG "resultType": "vector", 19:06:32 DEBUG "result": [ 19:06:32 DEBUG { 19:06:32 DEBUG "metric": { 19:06:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG [...] 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG 19:06:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:06:32 DEBUG }, 19:06:32 DEBUG "value": [ 19:06:32 DEBUG 1713204392, 19:06:32 DEBUG "413040.5" 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG ] 19:06:32 DEBUG } 19:06:32 DEBUG } 19:06:33 INFO 19:06:33 INFO [loop_until]: kubectl --namespace=xlou top pods 19:06:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:06:33 INFO [loop_until]: OK (rc = 0) 19:06:33 DEBUG --- stdout --- 19:06:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12735m 8983Mi am-7d597cf7d6-8vr8f 11719m 9007Mi am-7d597cf7d6-9wz9h 15612m 8822Mi ds-cts-0 15406m 8220Mi ds-cts-1 10891m 8406Mi ds-cts-2 10723m 8549Mi ds-idrepo-0 12863m 4928Mi ds-idrepo-1 2000m 4340Mi ds-idrepo-2 1884m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1022Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6718m 1474Mi 19:06:33 DEBUG --- stderr --- 19:06:33 DEBUG 19:06:33 INFO 19:06:33 INFO [loop_until]: kubectl --namespace=xlou top node 19:06:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:06:33 INFO [loop_until]: OK (rc = 0) 19:06:33 DEBUG --- stdout --- 19:06:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5105Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10835m 36% 10591Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13020m 43% 11055Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17635m 59% 3066Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2035m 6% 7132Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16345m 54% 12513Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2975Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12129m 40% 10534Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1999m 6% 6266Mi 5% gke-xlou-cdm-ds-0f601751-j488 10737m 35% 10099Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2442Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17187m 57% 10483Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13025m 43% 7083Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6938m 43% 5319Mi 9% 19:06:33 DEBUG --- stderr --- 19:06:33 DEBUG 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204452" 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204452" 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204452" 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204452" 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 ERROR [http_cmd]: ERROR 19:07:32 DEBUG --- status code --- 19:07:32 ERROR --- status code --- 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 ERROR --- http response --- 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:07:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG ----- output ----- 19:07:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG { 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:07:32 DEBUG "data": { 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG { 19:07:32 DEBUG { 19:07:32 DEBUG "result": [ 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG { 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG "data": { 19:07:32 DEBUG "data": { 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG "metric": { 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "resultType": "vector", 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG "pod": "ds-idrepo-1" 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG "result": [ 19:07:32 DEBUG "result": [ 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG }, 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG { 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG { 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG "value": [ 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG "metric": { 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG "metric": { 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:07:32 DEBUG 19:07:32 DEBUG { 19:07:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG "pod": "ds-cts-1" 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG }, 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG }, 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG { 19:07:32 DEBUG 19:07:32 DEBUG "data": { 19:07:32 DEBUG "value": [ 19:07:32 DEBUG { 19:07:32 DEBUG "value": [ 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:07:32 DEBUG { 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG [...] 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG 19:07:32 DEBUG "result": [ 19:07:32 DEBUG 19:07:32 DEBUG "data": { 19:07:32 DEBUG 19:07:32 DEBUG { 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG 19:07:32 DEBUG { 19:07:32 DEBUG 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG 19:07:32 DEBUG "metric": { 19:07:32 DEBUG [...] 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG "result": [ 19:07:32 DEBUG [...] 19:07:32 DEBUG "data": { 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "pod": "ds-idrepo-0" 19:07:32 DEBUG "instance": "10.142.0.107:9100" 19:07:32 DEBUG 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG { 19:07:32 DEBUG 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG }, 19:07:32 DEBUG }, 19:07:32 DEBUG 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG "metric": { 19:07:32 DEBUG 19:07:32 DEBUG "result": [ 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "pod": "ds-idrepo-1" 19:07:32 DEBUG "value": [ 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG "instance": "10.142.0.107:9100" 19:07:32 DEBUG 19:07:32 DEBUG { 19:07:32 DEBUG "pod": "ds-idrepo-1" 19:07:32 DEBUG }, 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG 19:07:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:07:32 DEBUG }, 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:07:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:07:32 DEBUG "metric": { 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG "0" 19:07:32 DEBUG 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG }, 19:07:32 DEBUG "instance": "10.142.0.107:9100" 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG ] 19:07:32 DEBUG 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG { 19:07:32 DEBUG "value": [ 19:07:32 DEBUG }, 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG } 19:07:32 DEBUG [...] 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG ] 19:07:32 DEBUG 19:07:32 DEBUG "2279.8" 19:07:32 DEBUG 19:07:32 DEBUG "data": { 19:07:32 DEBUG "0" 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG } 19:07:32 DEBUG 19:07:32 DEBUG ] 19:07:32 DEBUG [...] 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG ] 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG } 19:07:32 DEBUG 19:07:32 DEBUG } 19:07:32 DEBUG 19:07:32 DEBUG "result": [ 19:07:32 DEBUG } 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "instance": "10.142.0.43:9100" 19:07:32 DEBUG ] 19:07:32 DEBUG 19:07:32 DEBUG { 19:07:32 DEBUG ] 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG }, 19:07:32 DEBUG } 19:07:32 DEBUG 19:07:32 DEBUG "metric": { 19:07:32 DEBUG } 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "pod": "ds-idrepo-0" 19:07:32 DEBUG "value": [ 19:07:32 DEBUG } 19:07:32 DEBUG "instance": "10.142.0.43:9100" 19:07:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:07:32 DEBUG } 19:07:32 DEBUG 19:07:32 DEBUG "pod": "ds-cts-2" 19:07:32 DEBUG }, 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG }, 19:07:32 DEBUG }, 19:07:32 DEBUG 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG "5269.033333333333" 19:07:32 DEBUG "value": [ 19:07:32 DEBUG "value": [ 19:07:32 DEBUG "instance": "10.142.0.43:9100" 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG ] 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG 19:07:32 DEBUG }, 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "0" 19:07:32 DEBUG } 19:07:32 DEBUG "13777288192" 19:07:32 DEBUG 19:07:32 DEBUG "value": [ 19:07:32 DEBUG "0.00003333333333333331" 19:07:32 DEBUG ] 19:07:32 DEBUG ] 19:07:32 DEBUG ] 19:07:32 DEBUG 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 DEBUG } 19:07:32 DEBUG [...] 19:07:32 DEBUG "0" 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG 19:07:32 DEBUG ] 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 DEBUG 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 DEBUG 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:07:32 DEBUG } 19:07:32 DEBUG }, 19:07:32 DEBUG } 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "2290.733333333333" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG { 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "pod": "ds-cts-1" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "0" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG { 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "2280.9333333333334" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG { 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "0.078" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204452" 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204452" 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG { 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:07:32 DEBUG { 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG "metric": { 19:07:32 DEBUG { 19:07:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG }, 19:07:32 DEBUG "data": { 19:07:32 DEBUG "value": [ 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG 19:07:32 DEBUG "result": [ 19:07:32 DEBUG 19:07:32 DEBUG { 19:07:32 DEBUG 19:07:32 DEBUG "metric": { 19:07:32 DEBUG [...] 19:07:32 DEBUG "instance": "10.142.0.107:9100" 19:07:32 DEBUG 19:07:32 DEBUG }, 19:07:32 DEBUG 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:07:32 DEBUG 19:07:32 DEBUG }, 19:07:32 DEBUG 19:07:32 DEBUG "value": [ 19:07:32 DEBUG [...] 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG 19:07:32 DEBUG "6.656427781246044" 19:07:32 DEBUG 19:07:32 DEBUG ] 19:07:32 DEBUG 19:07:32 DEBUG } 19:07:32 DEBUG "instance": "10.142.0.43:9100" 19:07:32 DEBUG ] 19:07:32 DEBUG }, 19:07:32 DEBUG } 19:07:32 DEBUG "value": [ 19:07:32 DEBUG } 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "0.00026666666666651885" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204452" 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG { 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "instance": "10.142.0.107:9100" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "instance": "10.142.0.43:9100" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "5566.266666666667" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204452" 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG { 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "pod": "ds-idrepo-1" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "pod": "ds-idrepo-0" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "0" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG { 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "pod": "ds-idrepo-1" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "pod": "ds-idrepo-0" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "0" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG { 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "0" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG { 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "1642156032" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG { 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "pod": "ds-idrepo-1" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "pod": "ds-idrepo-0" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "0" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG { 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "pod": "ds-cts-1" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "0" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG { 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "pod": "ds-idrepo-1" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "pod": "ds-cts-2" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "0" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG { 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "pod": "ds-cts-1" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "0" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG { 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "2286.1" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204452" 19:07:32 INFO [http_cmd]: http status code OK 19:07:32 DEBUG --- status code --- 19:07:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:07:32 DEBUG --- http response --- 19:07:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:07:32 DEBUG ----- output ----- 19:07:32 DEBUG { 19:07:32 DEBUG "status": "success", 19:07:32 DEBUG "data": { 19:07:32 DEBUG "resultType": "vector", 19:07:32 DEBUG "result": [ 19:07:32 DEBUG { 19:07:32 DEBUG "metric": { 19:07:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG [...] 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG 19:07:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:07:32 DEBUG }, 19:07:32 DEBUG "value": [ 19:07:32 DEBUG 1713204452, 19:07:32 DEBUG "413106.5" 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG ] 19:07:32 DEBUG } 19:07:32 DEBUG } 19:07:33 INFO 19:07:33 INFO [loop_until]: kubectl --namespace=xlou top pods 19:07:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:07:33 INFO [loop_until]: OK (rc = 0) 19:07:33 DEBUG --- stdout --- 19:07:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12654m 8981Mi am-7d597cf7d6-8vr8f 11694m 9008Mi am-7d597cf7d6-9wz9h 15848m 8824Mi ds-cts-0 15182m 8218Mi ds-cts-1 11228m 8399Mi ds-cts-2 10389m 8577Mi ds-idrepo-0 12861m 4928Mi ds-idrepo-1 1982m 4341Mi ds-idrepo-2 1912m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6379m 1475Mi 19:07:33 DEBUG --- stderr --- 19:07:33 DEBUG 19:07:33 INFO 19:07:33 INFO [loop_until]: kubectl --namespace=xlou top node 19:07:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:07:33 INFO [loop_until]: OK (rc = 0) 19:07:33 DEBUG --- stdout --- 19:07:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5096Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10902m 36% 10600Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13071m 43% 11060Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17532m 58% 3062Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1997m 6% 7123Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16426m 55% 12520Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 66m 0% 2970Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12440m 41% 10533Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1987m 6% 6256Mi 5% gke-xlou-cdm-ds-0f601751-j488 10448m 34% 10103Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2428Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16977m 56% 10487Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12942m 43% 7083Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6819m 42% 5329Mi 9% 19:07:33 DEBUG --- stderr --- 19:07:33 DEBUG 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204512" 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 ERROR [http_cmd]: ERROR 19:08:32 ERROR --- status code --- 19:08:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:08:32 ERROR --- http response --- 19:08:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:08:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204512" 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "0" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "15.801830587458154" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "instance": "10.142.0.107:9100" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "instance": "10.142.0.43:9100" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "0" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204512" 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "pod": "ds-idrepo-1" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "ds-cts-2" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "0.00009612222222222222" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204512" 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 DEBUG --- status code --- 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:08:32 DEBUG { 19:08:32 DEBUG ----- output ----- 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG { 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "data": { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "data": { 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "result": [ 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "data": { 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG { 19:08:32 DEBUG "result": [ 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG "result": [ 19:08:32 DEBUG "metric": { 19:08:32 DEBUG { 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG { 19:08:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:08:32 DEBUG "metric": { 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "metric": { 19:08:32 DEBUG }, 19:08:32 DEBUG "instance": "10.142.0.107:9100" 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "data": { 19:08:32 DEBUG "pod": "ds-idrepo-1" 19:08:32 DEBUG "value": [ 19:08:32 DEBUG }, 19:08:32 DEBUG "pod": "ds-idrepo-1" 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG }, 19:08:32 DEBUG 19:08:32 DEBUG "value": [ 19:08:32 DEBUG }, 19:08:32 DEBUG "result": [ 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "value": [ 19:08:32 DEBUG { 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "metric": { 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG }, 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG }, 19:08:32 DEBUG "instance": "10.142.0.43:9100" 19:08:32 DEBUG 19:08:32 DEBUG "pod": "ds-idrepo-0" 19:08:32 DEBUG 19:08:32 DEBUG "value": [ 19:08:32 DEBUG }, 19:08:32 DEBUG "pod": "ds-idrepo-0" 19:08:32 DEBUG }, 19:08:32 DEBUG [...] 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG }, 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG "value": [ 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG 19:08:32 DEBUG "2294.866666666667" 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG 19:08:32 DEBUG ] 19:08:32 DEBUG "13778029568" 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG "0" 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG "0" 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG ] 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:08:32 DEBUG }, 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG "value": [ 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG { 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG { 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "2294.9" 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG } 19:08:32 DEBUG "data": { 19:08:32 DEBUG ] 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG } 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG ] 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG } 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG } 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:08:32 DEBUG "instance": "10.142.0.107:9100" 19:08:32 DEBUG }, 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:08:32 DEBUG "instance": "10.142.0.43:9100" 19:08:32 DEBUG }, 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "2292.6" 19:08:32 DEBUG "6914.7" 19:08:32 DEBUG ] 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204512" 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "instance": "10.142.0.107:9100" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "instance": "10.142.0.43:9100" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "0.0004666666666665264" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "0.078" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204512" 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "instance": "10.142.0.107:9100" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "instance": "10.142.0.43:9100" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "6017" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204512" 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "pod": "ds-idrepo-1" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "ds-idrepo-0" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "0" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "0" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "6.665765729405161" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204512" 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "pod": "ds-idrepo-1" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "ds-idrepo-0" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "0" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "46178.822680536345" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "pod": "ds-idrepo-1" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "ds-idrepo-0" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "0" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "1644122112" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "11739765.017136673" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "pod": "ds-idrepo-1" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "ds-cts-2" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "0" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "2294.7" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204512" 19:08:32 INFO [http_cmd]: http status code OK 19:08:32 DEBUG --- status code --- 19:08:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:08:32 DEBUG --- http response --- 19:08:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:08:32 DEBUG ----- output ----- 19:08:32 DEBUG { 19:08:32 DEBUG "status": "success", 19:08:32 DEBUG "data": { 19:08:32 DEBUG "resultType": "vector", 19:08:32 DEBUG "result": [ 19:08:32 DEBUG { 19:08:32 DEBUG "metric": { 19:08:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG [...] 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG 19:08:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:08:32 DEBUG }, 19:08:32 DEBUG "value": [ 19:08:32 DEBUG 1713204512, 19:08:32 DEBUG "412733" 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG ] 19:08:32 DEBUG } 19:08:32 DEBUG } 19:08:33 INFO 19:08:33 INFO [loop_until]: kubectl --namespace=xlou top pods 19:08:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:08:33 INFO [loop_until]: OK (rc = 0) 19:08:33 DEBUG --- stdout --- 19:08:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12770m 8978Mi am-7d597cf7d6-8vr8f 11607m 9006Mi am-7d597cf7d6-9wz9h 15865m 8826Mi ds-cts-0 15277m 8220Mi ds-cts-1 11022m 8408Mi ds-cts-2 10474m 8563Mi ds-idrepo-0 13105m 4928Mi ds-idrepo-1 2040m 4341Mi ds-idrepo-2 1913m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6716m 1477Mi 19:08:33 DEBUG --- stderr --- 19:08:33 DEBUG 19:08:33 INFO 19:08:33 INFO [loop_until]: kubectl --namespace=xlou top node 19:08:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:08:33 INFO [loop_until]: OK (rc = 0) 19:08:33 DEBUG --- stdout --- 19:08:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 5092Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10884m 36% 10605Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12980m 43% 11056Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17563m 58% 3061Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2017m 6% 7135Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 15874m 53% 12513Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2966Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12420m 41% 10521Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2000m 6% 6271Mi 5% gke-xlou-cdm-ds-0f601751-j488 10648m 35% 10087Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 66m 0% 2446Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16249m 54% 10487Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13022m 43% 7084Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6836m 43% 5328Mi 9% 19:08:33 DEBUG --- stderr --- 19:08:33 DEBUG 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 ERROR [http_cmd]: ERROR 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 ERROR --- status code --- 19:09:32 DEBUG --- status code --- 19:09:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 ERROR --- http response --- 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:09:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:09:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "0" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "instance": "10.142.0.107:9100" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "instance": "10.142.0.43:9100" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "0" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "0" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "ds-idrepo-1" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "ds-idrepo-0" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "0" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "ds-idrepo-1" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "ds-cts-2" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "0.000033333333333333335" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "ds-idrepo-1" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "ds-idrepo-0" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "0" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "instance": "10.142.0.107:9100" 19:09:32 DEBUG }, 19:09:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:09:32 DEBUG 19:09:32 DEBUG "instance": "10.142.0.43:9100" 19:09:32 DEBUG }, 19:09:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG }, 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "value": [ 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "2273.5666666666666" 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG "13778707456" 19:09:32 DEBUG ] 19:09:32 DEBUG "2273.233333333333" 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG } 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG } 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:09:32 DEBUG "instance": "10.142.0.107:9100" 19:09:32 DEBUG }, 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:09:32 DEBUG "instance": "10.142.0.43:9100" 19:09:32 DEBUG }, 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "2299.233333333333" 19:09:32 DEBUG "5307.966666666666" 19:09:32 DEBUG ] 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204572" 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "instance": "10.142.0.107:9100" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "instance": "10.142.0.43:9100" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "5726.333333333334" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204572" 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "instance": "10.142.0.107:9100" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "instance": "10.142.0.43:9100" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "0.00026666666666651885" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG { 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "6.720428851710612" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "0.078" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "ds-idrepo-1" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "ds-idrepo-0" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "0" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "0" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204572" 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "ds-idrepo-1" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "ds-idrepo-0" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "0" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "0" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "ds-idrepo-1" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "ds-idrepo-0" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "0" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "1648988160" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "551.5012791167362" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "ds-idrepo-1" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "ds-cts-2" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "0" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "2285.4" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204572" 19:09:32 INFO [http_cmd]: http status code OK 19:09:32 DEBUG --- status code --- 19:09:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:09:32 DEBUG --- http response --- 19:09:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:09:32 DEBUG ----- output ----- 19:09:32 DEBUG { 19:09:32 DEBUG "status": "success", 19:09:32 DEBUG "data": { 19:09:32 DEBUG "resultType": "vector", 19:09:32 DEBUG "result": [ 19:09:32 DEBUG { 19:09:32 DEBUG "metric": { 19:09:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG [...] 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG 19:09:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:09:32 DEBUG }, 19:09:32 DEBUG "value": [ 19:09:32 DEBUG 1713204572, 19:09:32 DEBUG "411965.5" 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG ] 19:09:32 DEBUG } 19:09:32 DEBUG } 19:09:33 INFO 19:09:33 INFO [loop_until]: kubectl --namespace=xlou top pods 19:09:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:09:33 INFO [loop_until]: OK (rc = 0) 19:09:33 DEBUG --- stdout --- 19:09:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12601m 8980Mi am-7d597cf7d6-8vr8f 11562m 9005Mi am-7d597cf7d6-9wz9h 15997m 8823Mi ds-cts-0 15630m 8219Mi ds-cts-1 10933m 8434Mi ds-cts-2 10787m 8562Mi ds-idrepo-0 13106m 4928Mi ds-idrepo-1 2115m 4340Mi ds-idrepo-2 1867m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1022Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6665m 1479Mi 19:09:33 DEBUG --- stderr --- 19:09:33 DEBUG 19:09:33 INFO 19:09:33 INFO [loop_until]: kubectl --namespace=xlou top node 19:09:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:09:34 INFO [loop_until]: OK (rc = 0) 19:09:34 DEBUG --- stdout --- 19:09:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5102Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11009m 36% 10616Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13054m 43% 11044Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 16943m 56% 3061Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1957m 6% 7126Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16589m 55% 12523Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2974Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12478m 41% 10530Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2036m 6% 6271Mi 5% gke-xlou-cdm-ds-0f601751-j488 10722m 35% 10102Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2438Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16979m 56% 10475Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13061m 43% 7080Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6863m 43% 5321Mi 9% 19:09:34 DEBUG --- stderr --- 19:09:34 DEBUG 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204632" 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 ERROR [http_cmd]: ERROR 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204632" 19:10:32 DEBUG --- status code --- 19:10:32 ERROR --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 ERROR --- http response --- 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:10:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:10:32 DEBUG ----- output ----- 19:10:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "pod": "ds-cts-1" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "0" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:10:32 DEBUG { 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG "metric": { 19:10:32 DEBUG { 19:10:32 DEBUG "pod": "ds-cts-1" 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG }, 19:10:32 DEBUG "data": { 19:10:32 DEBUG "value": [ 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG 19:10:32 DEBUG "result": [ 19:10:32 DEBUG 19:10:32 DEBUG { 19:10:32 DEBUG 19:10:32 DEBUG "metric": { 19:10:32 DEBUG [...] 19:10:32 DEBUG "pod": "ds-idrepo-1" 19:10:32 DEBUG 19:10:32 DEBUG }, 19:10:32 DEBUG 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:10:32 DEBUG 19:10:32 DEBUG }, 19:10:32 DEBUG 19:10:32 DEBUG "value": [ 19:10:32 DEBUG [...] 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG 19:10:32 DEBUG "0" 19:10:32 DEBUG 19:10:32 DEBUG ] 19:10:32 DEBUG 19:10:32 DEBUG } 19:10:32 DEBUG "pod": "ds-idrepo-0" 19:10:32 DEBUG ] 19:10:32 DEBUG }, 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG } 19:10:32 DEBUG "value": [ 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG } 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG "0" 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "instance": "10.142.0.107:9100" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "instance": "10.142.0.43:9100" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "0.00033333333333333365" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204632" 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "pod": "ds-idrepo-1" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "ds-cts-2" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "0.00008589111111111111" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "instance": "10.142.0.107:9100" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "instance": "10.142.0.43:9100" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "13779350528" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204632" 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "pod": "ds-idrepo-1" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "ds-idrepo-0" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "0" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "2268.366666666667" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "2268.0333333333333" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG { 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "2283.9" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "instance": "10.142.0.107:9100" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "instance": "10.142.0.43:9100" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "5065.166666666666" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204632" 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204632" 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "instance": "10.142.0.107:9100" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "instance": "10.142.0.43:9100" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "5052.200000000001" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "6.756756958991089" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204632" 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG { 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:10:32 DEBUG { 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG "metric": { 19:10:32 DEBUG { 19:10:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG }, 19:10:32 DEBUG "data": { 19:10:32 DEBUG "value": [ 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG 19:10:32 DEBUG "result": [ 19:10:32 DEBUG 19:10:32 DEBUG { 19:10:32 DEBUG 19:10:32 DEBUG "metric": { 19:10:32 DEBUG [...] 19:10:32 DEBUG "instance": "10.142.0.107:9100" 19:10:32 DEBUG 19:10:32 DEBUG }, 19:10:32 DEBUG 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:10:32 DEBUG 19:10:32 DEBUG }, 19:10:32 DEBUG 19:10:32 DEBUG "value": [ 19:10:32 DEBUG [...] 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG 19:10:32 DEBUG "0" 19:10:32 DEBUG 19:10:32 DEBUG ] 19:10:32 DEBUG 19:10:32 DEBUG } 19:10:32 DEBUG "instance": "10.142.0.43:9100" 19:10:32 DEBUG ] 19:10:32 DEBUG }, 19:10:32 DEBUG } 19:10:32 DEBUG "value": [ 19:10:32 DEBUG } 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "0.00030000000000001136" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "0.078" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "pod": "ds-idrepo-1" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "ds-idrepo-0" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "0" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204632" 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG { 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "pod": "ds-idrepo-1" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "ds-idrepo-0" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "0" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "pod": "ds-cts-1" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "0" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "pod": "ds-idrepo-1" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "ds-idrepo-0" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "0" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "1649123328" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG { 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "pod": "ds-idrepo-1" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "ds-cts-2" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "0" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "pod": "ds-cts-1" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "0" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204632" 19:10:32 INFO [http_cmd]: http status code OK 19:10:32 DEBUG --- status code --- 19:10:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:32 DEBUG --- http response --- 19:10:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:10:32 DEBUG ----- output ----- 19:10:32 DEBUG { 19:10:32 DEBUG "status": "success", 19:10:32 DEBUG "data": { 19:10:32 DEBUG "resultType": "vector", 19:10:32 DEBUG "result": [ 19:10:32 DEBUG { 19:10:32 DEBUG "metric": { 19:10:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG [...] 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG 19:10:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:10:32 DEBUG }, 19:10:32 DEBUG "value": [ 19:10:32 DEBUG 1713204632, 19:10:32 DEBUG "2268.366666666667" 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG ] 19:10:32 DEBUG } 19:10:32 DEBUG } 19:10:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204633" 19:10:33 INFO [http_cmd]: http status code OK 19:10:33 DEBUG --- status code --- 19:10:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:10:33 DEBUG --- http response --- 19:10:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:10:33 DEBUG ----- output ----- 19:10:33 DEBUG { 19:10:33 DEBUG "status": "success", 19:10:33 DEBUG "data": { 19:10:33 DEBUG "resultType": "vector", 19:10:33 DEBUG "result": [ 19:10:33 DEBUG { 19:10:33 DEBUG "metric": { 19:10:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:10:33 DEBUG }, 19:10:33 DEBUG "value": [ 19:10:33 DEBUG 19:10:33 DEBUG 19:10:33 DEBUG 19:10:33 DEBUG [...] 19:10:33 DEBUG 19:10:33 DEBUG 19:10:33 DEBUG 19:10:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:10:33 DEBUG }, 19:10:33 DEBUG "value": [ 19:10:33 DEBUG 1713204633, 19:10:33 DEBUG "411349.5" 19:10:33 DEBUG ] 19:10:33 DEBUG } 19:10:33 DEBUG ] 19:10:33 DEBUG } 19:10:33 DEBUG } 19:10:33 INFO 19:10:33 INFO [loop_until]: kubectl --namespace=xlou top pods 19:10:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:10:33 INFO [loop_until]: OK (rc = 0) 19:10:33 DEBUG --- stdout --- 19:10:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12639m 8979Mi am-7d597cf7d6-8vr8f 11538m 9008Mi am-7d597cf7d6-9wz9h 15949m 8823Mi ds-cts-0 15571m 8223Mi ds-cts-1 10861m 8417Mi ds-cts-2 10345m 8561Mi ds-idrepo-0 12758m 4930Mi ds-idrepo-1 1979m 4341Mi ds-idrepo-2 1916m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6851m 1480Mi 19:10:33 DEBUG --- stderr --- 19:10:33 DEBUG 19:10:34 INFO 19:10:34 INFO [loop_until]: kubectl --namespace=xlou top node 19:10:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:10:34 INFO [loop_until]: OK (rc = 0) 19:10:34 DEBUG --- stdout --- 19:10:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 5101Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10974m 36% 10614Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13076m 43% 11044Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17493m 58% 3065Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1982m 6% 7132Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16381m 54% 12540Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2983Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12461m 41% 10529Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2038m 6% 6277Mi 5% gke-xlou-cdm-ds-0f601751-j488 10448m 34% 10114Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2423Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 16805m 56% 10472Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13001m 43% 7077Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6934m 43% 5321Mi 9% 19:10:34 DEBUG --- stderr --- 19:10:34 DEBUG 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 ERROR [http_cmd]: ERROR 19:11:32 ERROR --- status code --- 19:11:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:11:32 ERROR --- http response --- 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG --- status code --- 19:11:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:11:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "pod": "ds-cts-1" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "0.00001542388735722175" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "pod": "ds-cts-1" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "0" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "instance": "10.142.0.107:9100" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "instance": "10.142.0.43:9100" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "0" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "pod": "ds-idrepo-1" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "pod": "ds-idrepo-0" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "0" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204692" 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "pod": "ds-idrepo-1" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "pod": "ds-cts-2" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "0.0001704622222222222" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:11:32 DEBUG "instance": "10.142.0.107:9100" 19:11:32 DEBUG }, 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 DEBUG "instance": "10.142.0.43:9100" 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG }, 19:11:32 DEBUG }, 19:11:32 DEBUG --- status code --- 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 DEBUG "value": [ 19:11:32 DEBUG "value": [ 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG "2299.2" 19:11:32 DEBUG "13779948544" 19:11:32 DEBUG ] 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG { 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 DEBUG "data": { 19:11:32 DEBUG } 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "instance": "10.142.0.107:9100" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "instance": "10.142.0.43:9100" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "5245.033333333334" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204692" 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "pod": "ds-idrepo-1" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "pod": "ds-idrepo-0" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "0" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "0" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "2298.1666666666665" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "2294.5666666666666" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG } 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "instance": "10.142.0.107:9100" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "instance": "10.142.0.43:9100" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "2915.7" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "0.078" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG { 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "instance": "10.142.0.107:9100" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "instance": "10.142.0.43:9100" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "0.00033333333333350387" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "6.81601811030589" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "pod": "ds-idrepo-1" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "pod": "ds-idrepo-0" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "0" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204692" 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "pod": "ds-idrepo-1" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "pod": "ds-idrepo-0" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "0" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "pod": "ds-idrepo-1" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "pod": "ds-idrepo-0" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "0" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "pod": "ds-cts-1" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "0" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:11:32 DEBUG ----- output ----- 19:11:32 DEBUG { 19:11:32 DEBUG "status": "success", 19:11:32 DEBUG "data": { 19:11:32 DEBUG "resultType": "vector", 19:11:32 DEBUG "result": [ 19:11:32 DEBUG { 19:11:32 DEBUG "metric": { 19:11:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG [...] 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG 19:11:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:11:32 DEBUG }, 19:11:32 DEBUG "value": [ 19:11:32 DEBUG 1713204692, 19:11:32 DEBUG "1651515392" 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG ] 19:11:32 DEBUG } 19:11:32 DEBUG } 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204692" 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 INFO [http_cmd]: http status code OK 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG --- status code --- 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:32 DEBUG --- http response --- 19:11:32 DEBUG --- http response --- 19:11:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:11:33 DEBUG ----- output ----- 19:11:33 DEBUG { 19:11:33 DEBUG "status": "success", 19:11:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:11:33 DEBUG "data": { 19:11:33 DEBUG ----- output ----- 19:11:33 DEBUG "resultType": "vector", 19:11:33 DEBUG { 19:11:33 DEBUG "result": [ 19:11:33 DEBUG "status": "success", 19:11:33 DEBUG { 19:11:33 DEBUG "data": { 19:11:33 DEBUG "metric": { 19:11:33 DEBUG "resultType": "vector", 19:11:33 DEBUG "pod": "ds-idrepo-1" 19:11:33 DEBUG "result": [ 19:11:33 DEBUG }, 19:11:33 DEBUG { 19:11:33 DEBUG "value": [ 19:11:33 DEBUG "metric": { 19:11:33 DEBUG 19:11:33 DEBUG "pod": "ds-cts-1" 19:11:33 DEBUG 19:11:33 DEBUG }, 19:11:33 DEBUG 19:11:33 DEBUG "value": [ 19:11:33 DEBUG [...] 19:11:33 DEBUG 19:11:33 DEBUG 19:11:33 DEBUG 19:11:33 DEBUG 19:11:33 DEBUG 19:11:33 DEBUG 19:11:33 DEBUG [...] 19:11:33 DEBUG "pod": "ds-cts-2" 19:11:33 DEBUG 19:11:33 DEBUG }, 19:11:33 DEBUG 19:11:33 DEBUG "value": [ 19:11:33 DEBUG 19:11:33 DEBUG 1713204692, 19:11:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:11:33 DEBUG "0" 19:11:33 DEBUG }, 19:11:33 DEBUG ] 19:11:33 DEBUG "value": [ 19:11:33 DEBUG } 19:11:33 DEBUG 1713204692, 19:11:33 DEBUG ] 19:11:33 DEBUG "0" 19:11:33 DEBUG } 19:11:33 DEBUG ] 19:11:33 DEBUG } 19:11:33 DEBUG } 19:11:33 DEBUG ] 19:11:33 DEBUG } 19:11:33 DEBUG } 19:11:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204693" 19:11:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204693" 19:11:33 INFO [http_cmd]: http status code OK 19:11:33 DEBUG --- status code --- 19:11:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:33 DEBUG --- http response --- 19:11:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:11:33 DEBUG ----- output ----- 19:11:33 DEBUG { 19:11:33 DEBUG "status": "success", 19:11:33 DEBUG "data": { 19:11:33 DEBUG "resultType": "vector", 19:11:33 DEBUG "result": [ 19:11:33 DEBUG { 19:11:33 DEBUG "metric": { 19:11:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:11:33 DEBUG }, 19:11:33 DEBUG "value": [ 19:11:33 DEBUG 19:11:33 DEBUG 19:11:33 DEBUG 19:11:33 DEBUG [...] 19:11:33 DEBUG 19:11:33 DEBUG 19:11:33 DEBUG 19:11:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:11:33 DEBUG }, 19:11:33 DEBUG "value": [ 19:11:33 DEBUG 1713204693, 19:11:33 DEBUG "2296.133333333333" 19:11:33 DEBUG ] 19:11:33 DEBUG } 19:11:33 DEBUG ] 19:11:33 DEBUG } 19:11:33 DEBUG } 19:11:33 INFO [http_cmd]: http status code OK 19:11:33 DEBUG --- status code --- 19:11:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:11:33 DEBUG --- http response --- 19:11:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:11:33 DEBUG ----- output ----- 19:11:33 DEBUG { 19:11:33 DEBUG "status": "success", 19:11:33 DEBUG "data": { 19:11:33 DEBUG "resultType": "vector", 19:11:33 DEBUG "result": [ 19:11:33 DEBUG { 19:11:33 DEBUG "metric": { 19:11:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:11:33 DEBUG }, 19:11:33 DEBUG "value": [ 19:11:33 DEBUG 19:11:33 DEBUG 19:11:33 DEBUG 19:11:33 DEBUG [...] 19:11:33 DEBUG 19:11:33 DEBUG 19:11:33 DEBUG 19:11:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:11:33 DEBUG }, 19:11:33 DEBUG "value": [ 19:11:33 DEBUG 1713204693, 19:11:33 DEBUG "411227" 19:11:33 DEBUG ] 19:11:33 DEBUG } 19:11:33 DEBUG ] 19:11:33 DEBUG } 19:11:33 DEBUG } 19:11:33 INFO 19:11:33 INFO [loop_until]: kubectl --namespace=xlou top pods 19:11:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:11:33 INFO [loop_until]: OK (rc = 0) 19:11:33 DEBUG --- stdout --- 19:11:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12565m 9000Mi am-7d597cf7d6-8vr8f 11449m 9008Mi am-7d597cf7d6-9wz9h 15653m 8822Mi ds-cts-0 15363m 8224Mi ds-cts-1 11378m 8417Mi ds-cts-2 10549m 8563Mi ds-idrepo-0 12987m 4930Mi ds-idrepo-1 1931m 4341Mi ds-idrepo-2 1976m 4112Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 5m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6819m 1482Mi 19:11:33 DEBUG --- stderr --- 19:11:33 DEBUG 19:11:34 INFO 19:11:34 INFO [loop_until]: kubectl --namespace=xlou top node 19:11:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:11:34 INFO [loop_until]: OK (rc = 0) 19:11:34 DEBUG --- stdout --- 19:11:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5103Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11062m 37% 10658Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12963m 43% 11061Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17521m 58% 3052Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1998m 6% 7130Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16507m 55% 12548Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2976Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12403m 41% 10529Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2046m 6% 6285Mi 5% gke-xlou-cdm-ds-0f601751-j488 10587m 35% 10110Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 59m 0% 2446Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17050m 57% 10479Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13035m 43% 7086Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 7007m 44% 5322Mi 9% 19:11:34 DEBUG --- stderr --- 19:11:34 DEBUG 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204752" 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG { 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG "data": { 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG "result": [ 19:12:32 DEBUG { 19:12:32 DEBUG "metric": { 19:12:32 DEBUG "pod": "ds-idrepo-1" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG [...] 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "pod": "ds-cts-2" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG "0.00009356444444444444" 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG } 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG { 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG "data": { 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG "result": [ 19:12:32 DEBUG { 19:12:32 DEBUG "metric": { 19:12:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG [...] 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "pod": "ds-cts-2" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG "10.444289218799758" 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG } 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG { 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG "data": { 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG "result": [ 19:12:32 DEBUG { 19:12:32 DEBUG "metric": { 19:12:32 DEBUG "instance": "10.142.0.107:9100" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG [...] 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "instance": "10.142.0.43:9100" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG "0.00033333333333333365" 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG } 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204752" 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 ERROR [http_cmd]: ERROR 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG --- status code --- 19:12:32 ERROR --- status code --- 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG --- http response --- 19:12:32 ERROR --- http response --- 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:12:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:12:32 DEBUG ----- output ----- 19:12:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG { 19:12:32 DEBUG { 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG { 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG "data": { 19:12:32 DEBUG "data": { 19:12:32 DEBUG "data": { 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG "result": [ 19:12:32 DEBUG "result": [ 19:12:32 DEBUG "result": [ 19:12:32 DEBUG { 19:12:32 DEBUG { 19:12:32 DEBUG { 19:12:32 DEBUG { 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG "metric": { 19:12:32 DEBUG "metric": { 19:12:32 DEBUG "metric": { 19:12:32 DEBUG "data": { 19:12:32 DEBUG "pod": "ds-idrepo-1" 19:12:32 DEBUG "instance": "10.142.0.107:9100" 19:12:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG }, 19:12:32 DEBUG }, 19:12:32 DEBUG }, 19:12:32 DEBUG "result": [ 19:12:32 DEBUG "value": [ 19:12:32 DEBUG "value": [ 19:12:32 DEBUG "value": [ 19:12:32 DEBUG { 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "metric": { 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG }, 19:12:32 DEBUG [...] 19:12:32 DEBUG [...] 19:12:32 DEBUG [...] 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "pod": "ds-idrepo-0" 19:12:32 DEBUG "instance": "10.142.0.43:9100" 19:12:32 DEBUG "pod": "ds-cts-2" 19:12:32 DEBUG [...] 19:12:32 DEBUG }, 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG }, 19:12:32 DEBUG }, 19:12:32 DEBUG 19:12:32 DEBUG "value": [ 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG "value": [ 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG 19:12:32 DEBUG "0" 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG "5194" 19:12:32 DEBUG "39692275.64306129" 19:12:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:12:32 DEBUG ] 19:12:32 DEBUG ] 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:12:32 DEBUG ] 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG }, 19:12:32 DEBUG } 19:12:32 DEBUG } 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG } 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG "value": [ 19:12:32 DEBUG ] 19:12:32 DEBUG ] 19:12:32 DEBUG { 19:12:32 DEBUG ] 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG } 19:12:32 DEBUG } 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG } 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG "2300.2" 19:12:32 DEBUG } 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG } 19:12:32 DEBUG "data": { 19:12:32 DEBUG } 19:12:32 DEBUG ] 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG } 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG "result": [ 19:12:32 DEBUG ] 19:12:32 DEBUG { 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG { 19:12:32 DEBUG } 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG "metric": { 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:12:32 DEBUG } 19:12:32 DEBUG "data": { 19:12:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG }, 19:12:32 DEBUG { 19:12:32 DEBUG "result": [ 19:12:32 DEBUG "value": [ 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG { 19:12:32 DEBUG 19:12:32 DEBUG "data": { 19:12:32 DEBUG "metric": { 19:12:32 DEBUG 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:12:32 DEBUG 19:12:32 DEBUG "result": [ 19:12:32 DEBUG }, 19:12:32 DEBUG [...] 19:12:32 DEBUG { 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 19:12:32 DEBUG "metric": { 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "instance": "10.142.0.107:9100" 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG }, 19:12:32 DEBUG 19:12:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:12:32 DEBUG "value": [ 19:12:32 DEBUG [...] 19:12:32 DEBUG }, 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "2301.0666666666666" 19:12:32 DEBUG [...] 19:12:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:12:32 DEBUG ] 19:12:32 DEBUG 19:12:32 DEBUG }, 19:12:32 DEBUG } 19:12:32 DEBUG 19:12:32 DEBUG "value": [ 19:12:32 DEBUG ] 19:12:32 DEBUG 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG } 19:12:32 DEBUG "instance": "10.142.0.43:9100" 19:12:32 DEBUG "0" 19:12:32 DEBUG } 19:12:32 DEBUG }, 19:12:32 DEBUG ] 19:12:32 DEBUG "value": [ 19:12:32 DEBUG } 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG ] 19:12:32 DEBUG "3099.9666666666667" 19:12:32 DEBUG } 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG } 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG } 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204752" 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG { 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG "data": { 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG "result": [ 19:12:32 DEBUG { 19:12:32 DEBUG "metric": { 19:12:32 DEBUG "instance": "10.142.0.107:9100" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG [...] 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "instance": "10.142.0.43:9100" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG "13780552704" 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG } 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG { 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG "data": { 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG "result": [ 19:12:32 DEBUG { 19:12:32 DEBUG "metric": { 19:12:32 DEBUG "pod": "ds-idrepo-1" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG [...] 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "pod": "ds-idrepo-0" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG "0" 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG } 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG { 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG "data": { 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG "result": [ 19:12:32 DEBUG { 19:12:32 DEBUG "metric": { 19:12:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG [...] 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG "2276.4333333333334" 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG } 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204752" 19:12:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204752" 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG { 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG "data": { 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG "result": [ 19:12:32 DEBUG { 19:12:32 DEBUG "metric": { 19:12:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG [...] 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG "0.078" 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG } 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG { 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG "data": { 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG "result": [ 19:12:32 DEBUG { 19:12:32 DEBUG "metric": { 19:12:32 DEBUG "pod": "ds-idrepo-1" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG [...] 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "pod": "ds-idrepo-0" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG "0" 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG } 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG { 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG "data": { 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG "result": [ 19:12:32 DEBUG { 19:12:32 DEBUG "metric": { 19:12:32 DEBUG "instance": "10.142.0.107:9100" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG [...] 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "instance": "10.142.0.43:9100" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG "0.00040000000000001514" 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG } 19:12:32 INFO [http_cmd]: http status code OK 19:12:32 DEBUG --- status code --- 19:12:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:32 DEBUG --- http response --- 19:12:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:12:32 DEBUG ----- output ----- 19:12:32 DEBUG { 19:12:32 DEBUG "status": "success", 19:12:32 DEBUG "data": { 19:12:32 DEBUG "resultType": "vector", 19:12:32 DEBUG "result": [ 19:12:32 DEBUG { 19:12:32 DEBUG "metric": { 19:12:32 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG [...] 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG 19:12:32 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:12:32 DEBUG }, 19:12:32 DEBUG "value": [ 19:12:32 DEBUG 1713204752, 19:12:32 DEBUG "6.7990020380316425" 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG ] 19:12:32 DEBUG } 19:12:32 DEBUG } 19:12:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204753" 19:12:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204753" 19:12:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204753" 19:12:33 INFO [http_cmd]: http status code OK 19:12:33 DEBUG --- status code --- 19:12:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:33 DEBUG --- http response --- 19:12:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:12:33 DEBUG ----- output ----- 19:12:33 DEBUG { 19:12:33 DEBUG "status": "success", 19:12:33 DEBUG "data": { 19:12:33 DEBUG "resultType": "vector", 19:12:33 DEBUG "result": [ 19:12:33 DEBUG { 19:12:33 DEBUG "metric": { 19:12:33 DEBUG "pod": "ds-idrepo-1" 19:12:33 DEBUG }, 19:12:33 DEBUG "value": [ 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG [...] 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG "pod": "ds-idrepo-0" 19:12:33 DEBUG }, 19:12:33 DEBUG "value": [ 19:12:33 DEBUG 1713204753, 19:12:33 DEBUG "0" 19:12:33 DEBUG ] 19:12:33 DEBUG } 19:12:33 DEBUG ] 19:12:33 DEBUG } 19:12:33 DEBUG } 19:12:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204753" 19:12:33 INFO [http_cmd]: http status code OK 19:12:33 DEBUG --- status code --- 19:12:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:33 DEBUG --- http response --- 19:12:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:12:33 DEBUG ----- output ----- 19:12:33 DEBUG { 19:12:33 DEBUG "status": "success", 19:12:33 DEBUG "data": { 19:12:33 DEBUG "resultType": "vector", 19:12:33 DEBUG "result": [ 19:12:33 DEBUG { 19:12:33 DEBUG "metric": { 19:12:33 INFO [http_cmd]: http status code OK 19:12:33 DEBUG "pod": "ds-idrepo-1" 19:12:33 DEBUG --- status code --- 19:12:33 DEBUG }, 19:12:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:33 DEBUG "value": [ 19:12:33 DEBUG --- http response --- 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG [...] 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG "pod": "ds-idrepo-0" 19:12:33 DEBUG }, 19:12:33 DEBUG "value": [ 19:12:33 DEBUG 1713204753, 19:12:33 DEBUG "0" 19:12:33 DEBUG ] 19:12:33 DEBUG } 19:12:33 DEBUG ] 19:12:33 DEBUG } 19:12:33 DEBUG } 19:12:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:12:33 DEBUG ----- output ----- 19:12:33 DEBUG { 19:12:33 DEBUG "status": "success", 19:12:33 DEBUG "data": { 19:12:33 DEBUG "resultType": "vector", 19:12:33 DEBUG "result": [ 19:12:33 DEBUG { 19:12:33 DEBUG "metric": { 19:12:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:12:33 DEBUG }, 19:12:33 DEBUG "value": [ 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG [...] 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG "pod": "ds-cts-2" 19:12:33 DEBUG }, 19:12:33 DEBUG "value": [ 19:12:33 DEBUG 1713204753, 19:12:33 DEBUG "220311.20990790727" 19:12:33 DEBUG ] 19:12:33 DEBUG } 19:12:33 DEBUG ] 19:12:33 DEBUG } 19:12:33 DEBUG } 19:12:33 INFO [http_cmd]: http status code OK 19:12:33 DEBUG --- status code --- 19:12:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:33 DEBUG --- http response --- 19:12:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:12:33 DEBUG ----- output ----- 19:12:33 DEBUG { 19:12:33 DEBUG "status": "success", 19:12:33 DEBUG "data": { 19:12:33 DEBUG "resultType": "vector", 19:12:33 DEBUG "result": [ 19:12:33 DEBUG { 19:12:33 DEBUG "metric": { 19:12:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:12:33 DEBUG }, 19:12:33 DEBUG "value": [ 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG [...] 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:12:33 DEBUG }, 19:12:33 DEBUG "value": [ 19:12:33 DEBUG 1713204753, 19:12:33 DEBUG "1650540544" 19:12:33 DEBUG ] 19:12:33 DEBUG } 19:12:33 DEBUG ] 19:12:33 DEBUG } 19:12:33 DEBUG } 19:12:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204753" 19:12:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204753" 19:12:33 INFO [http_cmd]: http status code OK 19:12:33 DEBUG --- status code --- 19:12:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:33 DEBUG --- http response --- 19:12:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:12:33 DEBUG ----- output ----- 19:12:33 DEBUG { 19:12:33 DEBUG "status": "success", 19:12:33 DEBUG "data": { 19:12:33 DEBUG "resultType": "vector", 19:12:33 DEBUG "result": [ 19:12:33 DEBUG { 19:12:33 DEBUG "metric": { 19:12:33 DEBUG "pod": "ds-idrepo-1" 19:12:33 INFO [http_cmd]: http status code OK 19:12:33 DEBUG }, 19:12:33 DEBUG --- status code --- 19:12:33 DEBUG "value": [ 19:12:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:33 DEBUG 19:12:33 DEBUG --- http response --- 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG [...] 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG "pod": "ds-cts-2" 19:12:33 DEBUG }, 19:12:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:12:33 DEBUG "value": [ 19:12:33 DEBUG ----- output ----- 19:12:33 DEBUG 1713204753, 19:12:33 DEBUG { 19:12:33 DEBUG "0" 19:12:33 DEBUG "status": "success", 19:12:33 DEBUG ] 19:12:33 DEBUG "data": { 19:12:33 DEBUG } 19:12:33 DEBUG "resultType": "vector", 19:12:33 DEBUG ] 19:12:33 DEBUG "result": [ 19:12:33 DEBUG } 19:12:33 DEBUG { 19:12:33 DEBUG } 19:12:33 DEBUG "metric": { 19:12:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:12:33 DEBUG }, 19:12:33 DEBUG "value": [ 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG [...] 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG "pod": "ds-cts-2" 19:12:33 DEBUG }, 19:12:33 DEBUG "value": [ 19:12:33 DEBUG 1713204753, 19:12:33 DEBUG "40483770.0857415" 19:12:33 DEBUG ] 19:12:33 DEBUG } 19:12:33 DEBUG ] 19:12:33 DEBUG } 19:12:33 DEBUG } 19:12:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204753" 19:12:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204753" 19:12:33 INFO [http_cmd]: http status code OK 19:12:33 DEBUG --- status code --- 19:12:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:33 DEBUG --- http response --- 19:12:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:12:33 DEBUG ----- output ----- 19:12:33 DEBUG { 19:12:33 DEBUG "status": "success", 19:12:33 DEBUG "data": { 19:12:33 DEBUG "resultType": "vector", 19:12:33 DEBUG "result": [ 19:12:33 DEBUG { 19:12:33 DEBUG "metric": { 19:12:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:12:33 DEBUG }, 19:12:33 DEBUG "value": [ 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG [...] 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:12:33 DEBUG }, 19:12:33 DEBUG "value": [ 19:12:33 DEBUG 1713204753, 19:12:33 DEBUG "2297.766666666667" 19:12:33 DEBUG ] 19:12:33 DEBUG } 19:12:33 DEBUG ] 19:12:33 DEBUG } 19:12:33 DEBUG } 19:12:33 INFO [http_cmd]: http status code OK 19:12:33 DEBUG --- status code --- 19:12:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:12:33 DEBUG --- http response --- 19:12:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:12:33 DEBUG ----- output ----- 19:12:33 DEBUG { 19:12:33 DEBUG "status": "success", 19:12:33 DEBUG "data": { 19:12:33 DEBUG "resultType": "vector", 19:12:33 DEBUG "result": [ 19:12:33 DEBUG { 19:12:33 DEBUG "metric": { 19:12:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:12:33 DEBUG }, 19:12:33 DEBUG "value": [ 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG [...] 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG 19:12:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:12:33 DEBUG }, 19:12:33 DEBUG "value": [ 19:12:33 DEBUG 1713204753, 19:12:33 DEBUG "411974" 19:12:33 DEBUG ] 19:12:33 DEBUG } 19:12:33 DEBUG ] 19:12:33 DEBUG } 19:12:33 DEBUG } 19:12:34 INFO 19:12:34 INFO [loop_until]: kubectl --namespace=xlou top pods 19:12:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:12:34 INFO [loop_until]: OK (rc = 0) 19:12:34 DEBUG --- stdout --- 19:12:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12575m 9071Mi am-7d597cf7d6-8vr8f 11419m 9009Mi am-7d597cf7d6-9wz9h 15538m 8822Mi ds-cts-0 15589m 8227Mi ds-cts-1 10801m 8418Mi ds-cts-2 10435m 8590Mi ds-idrepo-0 12589m 4929Mi ds-idrepo-1 2068m 4341Mi ds-idrepo-2 1881m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1022Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6667m 1484Mi 19:12:34 DEBUG --- stderr --- 19:12:34 DEBUG 19:12:34 INFO 19:12:34 INFO [loop_until]: kubectl --namespace=xlou top node 19:12:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:12:34 INFO [loop_until]: OK (rc = 0) 19:12:34 DEBUG --- stdout --- 19:12:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 59m 0% 5105Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10997m 36% 10690Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13048m 43% 11149Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17476m 58% 3045Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2019m 6% 7108Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16551m 55% 12535Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2981Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12379m 41% 10535Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1993m 6% 6262Mi 5% gke-xlou-cdm-ds-0f601751-j488 10470m 35% 10124Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2442Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17108m 57% 10478Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13009m 43% 7087Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6865m 43% 5320Mi 9% 19:12:34 DEBUG --- stderr --- 19:12:34 DEBUG 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204812" 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204812" 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 DEBUG --- status code --- 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:13:32 DEBUG ----- output ----- 19:13:32 DEBUG { 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG "data": { 19:13:32 DEBUG "resultType": "vector", 19:13:32 DEBUG "result": [ 19:13:32 DEBUG { 19:13:32 DEBUG "metric": { 19:13:32 DEBUG "pod": "ds-idrepo-1" 19:13:32 DEBUG }, 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG [...] 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG "pod": "ds-cts-2" 19:13:32 DEBUG }, 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 1713204812, 19:13:32 DEBUG "0.00022813555555555552" 19:13:32 DEBUG ] 19:13:32 DEBUG } 19:13:32 DEBUG ] 19:13:32 DEBUG } 19:13:32 DEBUG } 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 DEBUG --- status code --- 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:13:32 DEBUG ----- output ----- 19:13:32 DEBUG { 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG "data": { 19:13:32 DEBUG "resultType": "vector", 19:13:32 DEBUG "result": [ 19:13:32 DEBUG { 19:13:32 DEBUG "metric": { 19:13:32 DEBUG "instance": "10.142.0.107:9100" 19:13:32 DEBUG }, 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG [...] 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG "instance": "10.142.0.43:9100" 19:13:32 DEBUG }, 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 1713204812, 19:13:32 DEBUG "0" 19:13:32 DEBUG ] 19:13:32 DEBUG } 19:13:32 DEBUG ] 19:13:32 DEBUG } 19:13:32 DEBUG } 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204812" 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204812" 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204812" 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204812" 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204812" 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204812" 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204812" 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204812" 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 DEBUG --- status code --- 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204812" 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204812" 19:13:32 DEBUG --- status code --- 19:13:32 DEBUG --- status code --- 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 ERROR [http_cmd]: ERROR 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204812" 19:13:32 DEBUG --- status code --- 19:13:32 ERROR --- status code --- 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:13:32 ERROR --- http response --- 19:13:32 DEBUG --- status code --- 19:13:32 DEBUG --- status code --- 19:13:32 DEBUG ----- output ----- 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:13:32 DEBUG ----- output ----- 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 DEBUG { 19:13:32 DEBUG ----- output ----- 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 DEBUG { 19:13:32 DEBUG --- http response --- 19:13:32 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG { 19:13:32 DEBUG --- status code --- 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:13:32 DEBUG "data": { 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 DEBUG "data": { 19:13:32 DEBUG ----- output ----- 19:13:32 DEBUG "resultType": "vector", 19:13:32 DEBUG ----- output ----- 19:13:32 DEBUG "data": { 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG "resultType": "vector", 19:13:32 DEBUG { 19:13:32 DEBUG "result": [ 19:13:32 DEBUG { 19:13:32 DEBUG "resultType": "vector", 19:13:32 DEBUG "result": [ 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG { 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG "result": [ 19:13:32 DEBUG { 19:13:32 DEBUG ----- output ----- 19:13:32 DEBUG "data": { 19:13:32 DEBUG "metric": { 19:13:32 DEBUG "data": { 19:13:32 DEBUG { 19:13:32 DEBUG "metric": { 19:13:32 DEBUG { 19:13:32 DEBUG "resultType": "vector", 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:13:32 DEBUG "resultType": "vector", 19:13:32 DEBUG "metric": { 19:13:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG "result": [ 19:13:32 DEBUG --- status code --- 19:13:32 DEBUG --- status code --- 19:13:32 DEBUG }, 19:13:32 DEBUG "result": [ 19:13:32 DEBUG "instance": "10.142.0.107:9100" 19:13:32 DEBUG }, 19:13:32 DEBUG "data": { 19:13:32 DEBUG { 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 DEBUG "value": [ 19:13:32 DEBUG { 19:13:32 DEBUG }, 19:13:32 DEBUG "value": [ 19:13:32 DEBUG "resultType": "vector", 19:13:32 DEBUG "metric": { 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG 19:13:32 DEBUG "metric": { 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 19:13:32 DEBUG "result": [ 19:13:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:13:32 DEBUG 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:13:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG { 19:13:32 DEBUG }, 19:13:32 DEBUG ----- output ----- 19:13:32 DEBUG 19:13:32 DEBUG ----- output ----- 19:13:32 DEBUG }, 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG "metric": { 19:13:32 DEBUG "value": [ 19:13:32 DEBUG { 19:13:32 DEBUG [...] 19:13:32 DEBUG { 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 19:13:32 DEBUG [...] 19:13:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:13:32 DEBUG 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG 19:13:32 DEBUG [...] 19:13:32 DEBUG 19:13:32 DEBUG }, 19:13:32 DEBUG 19:13:32 DEBUG "data": { 19:13:32 DEBUG 19:13:32 DEBUG "data": { 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 19:13:32 DEBUG "resultType": "vector", 19:13:32 DEBUG 19:13:32 DEBUG "resultType": "vector", 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG [...] 19:13:32 DEBUG "result": [ 19:13:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:13:32 DEBUG "result": [ 19:13:32 DEBUG [...] 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 DEBUG 19:13:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG { 19:13:32 DEBUG }, 19:13:32 DEBUG { 19:13:32 DEBUG 19:13:32 DEBUG --- status code --- 19:13:32 DEBUG "instance": "10.142.0.43:9100" 19:13:32 DEBUG }, 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG "metric": { 19:13:32 DEBUG "value": [ 19:13:32 DEBUG "metric": { 19:13:32 DEBUG 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 DEBUG }, 19:13:32 DEBUG "value": [ 19:13:32 DEBUG [...] 19:13:32 DEBUG 19:13:32 DEBUG "pod": "ds-idrepo-1" 19:13:32 DEBUG 1713204812, 19:13:32 DEBUG "instance": "10.142.0.107:9100" 19:13:32 DEBUG 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 1713204812, 19:13:32 DEBUG 19:13:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:13:32 DEBUG }, 19:13:32 DEBUG "0" 19:13:32 DEBUG }, 19:13:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:13:32 DEBUG 1713204812, 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:13:32 DEBUG "15.857801880748674" 19:13:32 DEBUG 19:13:32 DEBUG }, 19:13:32 DEBUG "value": [ 19:13:32 DEBUG ] 19:13:32 DEBUG "value": [ 19:13:32 DEBUG }, 19:13:32 DEBUG "4661.133333333333" 19:13:32 DEBUG ----- output ----- 19:13:32 DEBUG ] 19:13:32 DEBUG 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 19:13:32 DEBUG } 19:13:32 DEBUG 19:13:32 DEBUG "value": [ 19:13:32 DEBUG ] 19:13:32 DEBUG { 19:13:32 DEBUG } 19:13:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:13:32 DEBUG 1713204812, 19:13:32 DEBUG 19:13:32 DEBUG ] 19:13:32 DEBUG 19:13:32 DEBUG 1713204812, 19:13:32 DEBUG } 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG ] 19:13:32 DEBUG }, 19:13:32 DEBUG "0" 19:13:32 DEBUG 19:13:32 DEBUG } 19:13:32 DEBUG 19:13:32 DEBUG "2267.2" 19:13:32 DEBUG ] 19:13:32 DEBUG "data": { 19:13:32 DEBUG } 19:13:32 DEBUG "value": [ 19:13:32 DEBUG ] 19:13:32 DEBUG [...] 19:13:32 DEBUG } 19:13:32 DEBUG [...] 19:13:32 DEBUG ] 19:13:32 DEBUG } 19:13:32 DEBUG "resultType": "vector", 19:13:32 DEBUG } 19:13:32 DEBUG 1713204812, 19:13:32 DEBUG } 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG } 19:13:32 DEBUG } 19:13:32 DEBUG "result": [ 19:13:32 DEBUG "2293.1" 19:13:32 DEBUG ] 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG ] 19:13:32 DEBUG { 19:13:32 DEBUG ] 19:13:32 DEBUG } 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG } 19:13:32 DEBUG "metric": { 19:13:32 DEBUG } 19:13:32 DEBUG } 19:13:32 DEBUG "pod": "ds-idrepo-0" 19:13:32 DEBUG "instance": "10.142.0.43:9100" 19:13:32 DEBUG } 19:13:32 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:13:32 DEBUG ] 19:13:32 DEBUG }, 19:13:32 DEBUG }, 19:13:32 DEBUG }, 19:13:32 DEBUG } 19:13:32 DEBUG "value": [ 19:13:32 DEBUG "value": [ 19:13:32 DEBUG "value": [ 19:13:32 DEBUG } 19:13:32 DEBUG 1713204812, 19:13:32 DEBUG 1713204812, 19:13:32 DEBUG 19:13:32 DEBUG "0" 19:13:32 DEBUG "6039.766666666667" 19:13:32 DEBUG 19:13:32 DEBUG ] 19:13:32 DEBUG ] 19:13:32 DEBUG 19:13:32 DEBUG } 19:13:32 DEBUG } 19:13:32 DEBUG [...] 19:13:32 DEBUG ] 19:13:32 DEBUG ] 19:13:32 DEBUG 19:13:32 DEBUG } 19:13:32 DEBUG } 19:13:32 DEBUG 19:13:32 DEBUG } 19:13:32 DEBUG } 19:13:32 DEBUG 19:13:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:13:32 DEBUG }, 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 1713204812, 19:13:32 DEBUG "2292.9666666666667" 19:13:32 DEBUG ] 19:13:32 DEBUG } 19:13:32 DEBUG ] 19:13:32 DEBUG } 19:13:32 DEBUG } 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204812" 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204812" 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204812" 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 DEBUG --- status code --- 19:13:32 DEBUG --- status code --- 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:13:32 DEBUG ----- output ----- 19:13:32 DEBUG { 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG "data": { 19:13:32 DEBUG "resultType": "vector", 19:13:32 DEBUG "result": [ 19:13:32 DEBUG { 19:13:32 DEBUG "metric": { 19:13:32 DEBUG "instance": "10.142.0.107:9100" 19:13:32 DEBUG }, 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG [...] 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG "instance": "10.142.0.43:9100" 19:13:32 DEBUG }, 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 1713204812, 19:13:32 DEBUG "13781371904" 19:13:32 DEBUG ] 19:13:32 DEBUG } 19:13:32 DEBUG ] 19:13:32 DEBUG } 19:13:32 DEBUG } 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:13:32 DEBUG ----- output ----- 19:13:32 DEBUG { 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG "data": { 19:13:32 DEBUG "resultType": "vector", 19:13:32 DEBUG "result": [ 19:13:32 DEBUG { 19:13:32 DEBUG "metric": { 19:13:32 DEBUG "pod": "ds-idrepo-1" 19:13:32 DEBUG }, 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG [...] 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG "pod": "ds-idrepo-0" 19:13:32 DEBUG }, 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 1713204812, 19:13:32 DEBUG "0" 19:13:32 DEBUG ] 19:13:32 DEBUG } 19:13:32 DEBUG ] 19:13:32 DEBUG } 19:13:32 DEBUG } 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 DEBUG --- status code --- 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:13:32 DEBUG ----- output ----- 19:13:32 DEBUG { 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG "data": { 19:13:32 DEBUG "resultType": "vector", 19:13:32 DEBUG "result": [ 19:13:32 DEBUG { 19:13:32 DEBUG "metric": { 19:13:32 DEBUG "instance": "10.142.0.107:9100" 19:13:32 DEBUG }, 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG [...] 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG 19:13:32 DEBUG "instance": "10.142.0.43:9100" 19:13:32 DEBUG }, 19:13:32 DEBUG "value": [ 19:13:32 DEBUG 1713204812, 19:13:32 DEBUG "0.00043333333333350765" 19:13:32 DEBUG ] 19:13:32 DEBUG } 19:13:32 DEBUG ] 19:13:32 DEBUG } 19:13:32 DEBUG } 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204812" 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204812" 19:13:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204812" 19:13:32 INFO [http_cmd]: http status code OK 19:13:32 DEBUG --- status code --- 19:13:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:32 DEBUG --- http response --- 19:13:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:13:32 DEBUG ----- output ----- 19:13:32 DEBUG { 19:13:32 DEBUG "status": "success", 19:13:32 DEBUG "data": { 19:13:32 DEBUG "resultType": "vector", 19:13:32 DEBUG "result": [ 19:13:32 DEBUG { 19:13:32 DEBUG "metric": { 19:13:32 DEBUG "pod": "ds-idrepo-1" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG [...] 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG "pod": "ds-idrepo-0" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 1713204812, 19:13:33 DEBUG "0" 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG } 19:13:33 INFO [http_cmd]: http status code OK 19:13:33 DEBUG --- status code --- 19:13:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:33 DEBUG --- http response --- 19:13:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:13:33 DEBUG ----- output ----- 19:13:33 DEBUG { 19:13:33 DEBUG "status": "success", 19:13:33 DEBUG "data": { 19:13:33 DEBUG "resultType": "vector", 19:13:33 DEBUG "result": [ 19:13:33 DEBUG { 19:13:33 DEBUG "metric": { 19:13:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG [...] 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 1713204812, 19:13:33 DEBUG "0.078" 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG } 19:13:33 INFO [http_cmd]: http status code OK 19:13:33 DEBUG --- status code --- 19:13:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:33 DEBUG --- http response --- 19:13:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:13:33 DEBUG ----- output ----- 19:13:33 DEBUG { 19:13:33 DEBUG "status": "success", 19:13:33 DEBUG "data": { 19:13:33 DEBUG "resultType": "vector", 19:13:33 DEBUG "result": [ 19:13:33 DEBUG { 19:13:33 DEBUG "metric": { 19:13:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG [...] 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 1713204812, 19:13:33 DEBUG "6.68553648875202" 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG } 19:13:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204813" 19:13:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204813" 19:13:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204813" 19:13:33 INFO [http_cmd]: http status code OK 19:13:33 DEBUG --- status code --- 19:13:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:33 DEBUG --- http response --- 19:13:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:13:33 DEBUG ----- output ----- 19:13:33 DEBUG { 19:13:33 DEBUG "status": "success", 19:13:33 DEBUG "data": { 19:13:33 DEBUG "resultType": "vector", 19:13:33 DEBUG "result": [ 19:13:33 DEBUG { 19:13:33 DEBUG "metric": { 19:13:33 DEBUG "pod": "ds-idrepo-1" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG [...] 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG "pod": "ds-idrepo-0" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 1713204813, 19:13:33 DEBUG "0" 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG } 19:13:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204813" 19:13:33 INFO [http_cmd]: http status code OK 19:13:33 DEBUG --- status code --- 19:13:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:33 DEBUG --- http response --- 19:13:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:13:33 DEBUG ----- output ----- 19:13:33 DEBUG { 19:13:33 DEBUG "status": "success", 19:13:33 DEBUG "data": { 19:13:33 DEBUG "resultType": "vector", 19:13:33 DEBUG "result": [ 19:13:33 DEBUG { 19:13:33 DEBUG "metric": { 19:13:33 DEBUG "pod": "ds-idrepo-1" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG [...] 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG "pod": "ds-idrepo-0" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 1713204813, 19:13:33 DEBUG "0" 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG } 19:13:33 INFO [http_cmd]: http status code OK 19:13:33 DEBUG --- status code --- 19:13:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:33 DEBUG --- http response --- 19:13:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:13:33 DEBUG ----- output ----- 19:13:33 DEBUG { 19:13:33 DEBUG "status": "success", 19:13:33 DEBUG "data": { 19:13:33 DEBUG "resultType": "vector", 19:13:33 DEBUG "result": [ 19:13:33 DEBUG { 19:13:33 DEBUG "metric": { 19:13:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG [...] 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 1713204813, 19:13:33 DEBUG "269437212.5305695" 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG } 19:13:33 INFO [http_cmd]: http status code OK 19:13:33 DEBUG --- status code --- 19:13:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:33 DEBUG --- http response --- 19:13:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:13:33 DEBUG ----- output ----- 19:13:33 DEBUG { 19:13:33 DEBUG "status": "success", 19:13:33 DEBUG "data": { 19:13:33 DEBUG "resultType": "vector", 19:13:33 DEBUG "result": [ 19:13:33 DEBUG { 19:13:33 DEBUG "metric": { 19:13:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG [...] 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 1713204813, 19:13:33 DEBUG "1653186560" 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG } 19:13:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204813" 19:13:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204813" 19:13:33 INFO [http_cmd]: http status code OK 19:13:33 DEBUG --- status code --- 19:13:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:33 DEBUG --- http response --- 19:13:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:13:33 DEBUG ----- output ----- 19:13:33 DEBUG { 19:13:33 DEBUG "status": "success", 19:13:33 DEBUG "data": { 19:13:33 DEBUG "resultType": "vector", 19:13:33 DEBUG "result": [ 19:13:33 DEBUG { 19:13:33 DEBUG "metric": { 19:13:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG [...] 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 1713204813, 19:13:33 DEBUG "10456319.552812392" 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG } 19:13:33 INFO [http_cmd]: http status code OK 19:13:33 DEBUG --- status code --- 19:13:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:33 DEBUG --- http response --- 19:13:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:13:33 DEBUG ----- output ----- 19:13:33 DEBUG { 19:13:33 DEBUG "status": "success", 19:13:33 DEBUG "data": { 19:13:33 DEBUG "resultType": "vector", 19:13:33 DEBUG "result": [ 19:13:33 DEBUG { 19:13:33 DEBUG "metric": { 19:13:33 DEBUG "pod": "ds-idrepo-1" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG [...] 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG "pod": "ds-cts-2" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 1713204813, 19:13:33 DEBUG "0" 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG } 19:13:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204813" 19:13:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204813" 19:13:33 INFO [http_cmd]: http status code OK 19:13:33 DEBUG --- status code --- 19:13:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:33 DEBUG --- http response --- 19:13:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:13:33 DEBUG ----- output ----- 19:13:33 DEBUG { 19:13:33 DEBUG "status": "success", 19:13:33 DEBUG "data": { 19:13:33 DEBUG "resultType": "vector", 19:13:33 DEBUG "result": [ 19:13:33 DEBUG { 19:13:33 DEBUG "metric": { 19:13:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG [...] 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 1713204813, 19:13:33 DEBUG "2296.9333333333334" 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG } 19:13:33 INFO [http_cmd]: http status code OK 19:13:33 DEBUG --- status code --- 19:13:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:13:33 DEBUG --- http response --- 19:13:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:13:33 DEBUG ----- output ----- 19:13:33 DEBUG { 19:13:33 DEBUG "status": "success", 19:13:33 DEBUG "data": { 19:13:33 DEBUG "resultType": "vector", 19:13:33 DEBUG "result": [ 19:13:33 DEBUG { 19:13:33 DEBUG "metric": { 19:13:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG [...] 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG 19:13:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:13:33 DEBUG }, 19:13:33 DEBUG "value": [ 19:13:33 DEBUG 1713204813, 19:13:33 DEBUG "412688.5" 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG ] 19:13:33 DEBUG } 19:13:33 DEBUG } 19:13:34 INFO 19:13:34 INFO [loop_until]: kubectl --namespace=xlou top pods 19:13:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:13:34 INFO [loop_until]: OK (rc = 0) 19:13:34 DEBUG --- stdout --- 19:13:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12721m 9010Mi am-7d597cf7d6-8vr8f 11414m 9008Mi am-7d597cf7d6-9wz9h 15741m 8827Mi ds-cts-0 16113m 8209Mi ds-cts-1 10779m 8429Mi ds-cts-2 10749m 8583Mi ds-idrepo-0 12837m 4942Mi ds-idrepo-1 2035m 4341Mi ds-idrepo-2 1908m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6584m 1484Mi 19:13:34 DEBUG --- stderr --- 19:13:34 DEBUG 19:13:34 INFO 19:13:34 INFO [loop_until]: kubectl --namespace=xlou top node 19:13:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:13:34 INFO [loop_until]: OK (rc = 0) 19:13:34 DEBUG --- stdout --- 19:13:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5102Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10975m 36% 10656Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13113m 43% 11100Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 16960m 56% 3058Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2003m 6% 7108Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16592m 55% 12481Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2972Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12391m 41% 10531Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2006m 6% 6272Mi 5% gke-xlou-cdm-ds-0f601751-j488 10668m 35% 10116Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2453Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17004m 56% 10478Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13009m 43% 7085Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6593m 41% 5320Mi 9% 19:13:34 DEBUG --- stderr --- 19:13:34 DEBUG 19:14:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204872" 19:14:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204872" 19:14:32 INFO [http_cmd]: http status code OK 19:14:32 DEBUG --- status code --- 19:14:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:32 DEBUG --- http response --- 19:14:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:14:32 DEBUG ----- output ----- 19:14:32 DEBUG { 19:14:32 DEBUG "status": "success", 19:14:32 DEBUG "data": { 19:14:32 DEBUG "resultType": "vector", 19:14:32 DEBUG "result": [ 19:14:32 DEBUG { 19:14:32 DEBUG "metric": { 19:14:32 DEBUG "pod": "ds-idrepo-1" 19:14:32 DEBUG }, 19:14:32 DEBUG "value": [ 19:14:32 DEBUG 19:14:32 DEBUG 19:14:32 DEBUG 19:14:32 DEBUG [...] 19:14:32 DEBUG 19:14:32 DEBUG 19:14:32 DEBUG 19:14:32 DEBUG "pod": "ds-cts-2" 19:14:32 DEBUG }, 19:14:32 DEBUG "value": [ 19:14:32 DEBUG 1713204872, 19:14:32 DEBUG "0.00010000000000000009" 19:14:32 DEBUG ] 19:14:32 DEBUG } 19:14:32 DEBUG ] 19:14:32 DEBUG } 19:14:32 DEBUG } 19:14:32 INFO [http_cmd]: http status code OK 19:14:32 DEBUG --- status code --- 19:14:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:32 DEBUG --- http response --- 19:14:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:14:32 DEBUG ----- output ----- 19:14:32 DEBUG { 19:14:32 DEBUG "status": "success", 19:14:32 DEBUG "data": { 19:14:32 DEBUG "resultType": "vector", 19:14:32 DEBUG "result": [ 19:14:32 DEBUG { 19:14:32 DEBUG "metric": { 19:14:32 DEBUG "instance": "10.142.0.107:9100" 19:14:32 DEBUG }, 19:14:32 DEBUG "value": [ 19:14:32 DEBUG 19:14:32 DEBUG 19:14:32 DEBUG 19:14:32 DEBUG [...] 19:14:32 DEBUG 19:14:32 DEBUG 19:14:32 DEBUG 19:14:32 DEBUG "instance": "10.142.0.43:9100" 19:14:32 DEBUG }, 19:14:32 DEBUG "value": [ 19:14:32 DEBUG 1713204872, 19:14:32 DEBUG "0" 19:14:32 DEBUG ] 19:14:32 DEBUG } 19:14:32 DEBUG ] 19:14:32 DEBUG } 19:14:32 DEBUG } 19:14:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204872" 19:14:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204872" 19:14:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204872" 19:14:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204872" 19:14:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204872" 19:14:32 INFO [http_cmd]: http status code OK 19:14:32 INFO [http_cmd]: http status code OK 19:14:32 DEBUG --- status code --- 19:14:32 DEBUG --- status code --- 19:14:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:32 DEBUG --- http response --- 19:14:32 DEBUG --- http response --- 19:14:32 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:14:32 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:14:32 DEBUG ----- output ----- 19:14:32 DEBUG { 19:14:32 DEBUG "status": "success", 19:14:32 DEBUG "data": { 19:14:32 DEBUG "resultType": "vector", 19:14:32 DEBUG "result": [ 19:14:32 DEBUG { 19:14:32 DEBUG "metric": { 19:14:32 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:14:32 DEBUG }, 19:14:32 DEBUG "value": [ 19:14:32 DEBUG 19:14:32 DEBUG 19:14:32 DEBUG 19:14:32 DEBUG [...] 19:14:32 DEBUG 19:14:32 DEBUG 19:14:32 DEBUG 19:14:32 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:14:32 DEBUG }, 19:14:32 DEBUG "value": [ 19:14:32 DEBUG 1713204872, 19:14:32 DEBUG "0" 19:14:32 DEBUG ] 19:14:32 DEBUG } 19:14:32 DEBUG ] 19:14:32 DEBUG } 19:14:32 DEBUG } 19:14:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204872" 19:14:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204872" 19:14:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204872" 19:14:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204872" 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204873" 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204873" 19:14:33 ERROR [http_cmd]: ERROR 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204873" 19:14:33 ERROR --- status code --- 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG --- status code --- 19:14:33 ERROR --- http response --- 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- http response --- 19:14:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG { 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "data": { 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "result": [ 19:14:33 DEBUG { 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 1713204872, 19:14:33 DEBUG "2308" 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:14:33 DEBUG --- status code --- 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG --- status code --- 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG { 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:14:33 DEBUG "data": { 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:14:33 DEBUG ----- output ----- 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:14:33 DEBUG "resultType": "vector", 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG { 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG { 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG "result": [ 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG { 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG { 19:14:33 DEBUG { 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG "data": { 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "data": { 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "metric": { 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG "data": { 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "data": { 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:14:33 DEBUG "instance": "10.142.0.107:9100" 19:14:33 DEBUG "result": [ 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:14:33 DEBUG "result": [ 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG }, 19:14:33 DEBUG { 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG "result": [ 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG { 19:14:33 DEBUG "result": [ 19:14:33 DEBUG { 19:14:33 DEBUG "value": [ 19:14:33 DEBUG "metric": { 19:14:33 DEBUG { 19:14:33 DEBUG { 19:14:33 DEBUG { 19:14:33 DEBUG "metric": { 19:14:33 DEBUG { 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG 19:14:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "data": { 19:14:33 DEBUG 19:14:33 DEBUG }, 19:14:33 DEBUG "data": { 19:14:33 DEBUG "instance": "10.142.0.107:9100" 19:14:33 DEBUG "data": { 19:14:33 DEBUG }, 19:14:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG 19:14:33 DEBUG "value": [ 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG }, 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "value": [ 19:14:33 DEBUG }, 19:14:33 DEBUG "result": [ 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG "result": [ 19:14:33 DEBUG "value": [ 19:14:33 DEBUG "result": [ 19:14:33 DEBUG 19:14:33 DEBUG "value": [ 19:14:33 DEBUG { 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG { 19:14:33 DEBUG 19:14:33 DEBUG { 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "metric": { 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "metric": { 19:14:33 DEBUG 19:14:33 DEBUG "metric": { 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG "pod": "ds-idrepo-1" 19:14:33 DEBUG 19:14:33 DEBUG "instance": "10.142.0.107:9100" 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG }, 19:14:33 DEBUG "instance": "10.142.0.43:9100" 19:14:33 DEBUG 19:14:33 DEBUG }, 19:14:33 DEBUG [...] 19:14:33 DEBUG }, 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG "value": [ 19:14:33 DEBUG }, 19:14:33 DEBUG 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 1713204872, 19:14:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "13782053888" 19:14:33 DEBUG }, 19:14:33 DEBUG 19:14:33 DEBUG "instance": "10.142.0.43:9100" 19:14:33 DEBUG 19:14:33 DEBUG }, 19:14:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:14:33 DEBUG [...] 19:14:33 DEBUG ] 19:14:33 DEBUG "value": [ 19:14:33 DEBUG [...] 19:14:33 DEBUG }, 19:14:33 DEBUG [...] 19:14:33 DEBUG "value": [ 19:14:33 DEBUG }, 19:14:33 DEBUG 19:14:33 DEBUG } 19:14:33 DEBUG 1713204872, 19:14:33 DEBUG 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 1713204872, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG ] 19:14:33 DEBUG "0" 19:14:33 DEBUG 19:14:33 DEBUG 1713204872, 19:14:33 DEBUG 19:14:33 DEBUG "2299.2" 19:14:33 DEBUG 1713204872, 19:14:33 DEBUG 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG 19:14:33 DEBUG "3073.4" 19:14:33 DEBUG 19:14:33 DEBUG ] 19:14:33 DEBUG "9.03919270126463" 19:14:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 DEBUG "pod": "ds-idrepo-0" 19:14:33 DEBUG ] 19:14:33 DEBUG "instance": "10.142.0.43:9100" 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG }, 19:14:33 DEBUG ] 19:14:33 DEBUG }, 19:14:33 DEBUG } 19:14:33 DEBUG }, 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG "value": [ 19:14:33 DEBUG } 19:14:33 DEBUG "value": [ 19:14:33 DEBUG ] 19:14:33 DEBUG "value": [ 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG } 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG } 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 DEBUG "2308.2" 19:14:33 DEBUG "0" 19:14:33 DEBUG } 19:14:33 DEBUG "5127.0666666666675" 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG ] 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204873" 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG ] 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204873" 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204873" 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG { 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "data": { 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "result": [ 19:14:33 DEBUG { 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "pod": "ds-idrepo-1" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "pod": "ds-idrepo-0" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG "0" 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG { 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "data": { 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "result": [ 19:14:33 DEBUG { 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "instance": "10.142.0.107:9100" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "instance": "10.142.0.43:9100" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG "0.00023333333333302637" 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG { 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "data": { 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "result": [ 19:14:33 DEBUG { 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG "6.719396854079392" 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204873" 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204873" 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG { 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "data": { 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "result": [ 19:14:33 DEBUG { 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "pod": "ds-idrepo-1" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "pod": "ds-idrepo-0" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG "0" 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG { 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "data": { 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "result": [ 19:14:33 DEBUG { 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG "0.078" 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204873" 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG { 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "data": { 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "result": [ 19:14:33 DEBUG { 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "pod": "ds-idrepo-1" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "pod": "ds-idrepo-0" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG "0" 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204873" 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204873" 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204873" 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204873" 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG --- status code --- 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:14:33 DEBUG { 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG { 19:14:33 DEBUG "data": { 19:14:33 DEBUG { 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "data": { 19:14:33 DEBUG "result": [ 19:14:33 DEBUG "data": { 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG { 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "result": [ 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "result": [ 19:14:33 DEBUG { 19:14:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:14:33 DEBUG { 19:14:33 DEBUG "metric": { 19:14:33 DEBUG }, 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "pod": "ds-idrepo-1" 19:14:33 DEBUG "value": [ 19:14:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:14:33 DEBUG }, 19:14:33 DEBUG 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG }, 19:14:33 DEBUG 19:14:33 DEBUG "pod": "ds-idrepo-0" 19:14:33 DEBUG "value": [ 19:14:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:14:33 DEBUG }, 19:14:33 DEBUG 1713204873, 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG "3294923.3830577116" 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG ] 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG "0" 19:14:33 DEBUG } 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG "129904.6238722016" 19:14:33 DEBUG ] 19:14:33 DEBUG ] 19:14:33 DEBUG ] 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG { 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 DEBUG "data": { 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "result": [ 19:14:33 DEBUG { 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG "1652621312" 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204873" 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG { 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "data": { 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "result": [ 19:14:33 DEBUG { 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "pod": "ds-idrepo-1" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "pod": "ds-cts-2" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG "0" 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204873" 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG { 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "data": { 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "result": [ 19:14:33 DEBUG { 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG "2300.233333333333" 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204873" 19:14:33 INFO [http_cmd]: http status code OK 19:14:33 DEBUG --- status code --- 19:14:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:14:33 DEBUG --- http response --- 19:14:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:14:33 DEBUG ----- output ----- 19:14:33 DEBUG { 19:14:33 DEBUG "status": "success", 19:14:33 DEBUG "data": { 19:14:33 DEBUG "resultType": "vector", 19:14:33 DEBUG "result": [ 19:14:33 DEBUG { 19:14:33 DEBUG "metric": { 19:14:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG [...] 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG 19:14:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:14:33 DEBUG }, 19:14:33 DEBUG "value": [ 19:14:33 DEBUG 1713204873, 19:14:33 DEBUG "413245.5" 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG ] 19:14:33 DEBUG } 19:14:33 DEBUG } 19:14:34 INFO 19:14:34 INFO [loop_until]: kubectl --namespace=xlou top pods 19:14:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:14:34 INFO [loop_until]: OK (rc = 0) 19:14:34 DEBUG --- stdout --- 19:14:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12729m 9011Mi am-7d597cf7d6-8vr8f 11967m 9008Mi am-7d597cf7d6-9wz9h 15593m 8829Mi ds-cts-0 15626m 8194Mi ds-cts-1 10770m 8436Mi ds-cts-2 10445m 8583Mi ds-idrepo-0 13167m 4941Mi ds-idrepo-1 1974m 4341Mi ds-idrepo-2 1892m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 4m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6700m 1486Mi 19:14:34 DEBUG --- stderr --- 19:14:34 DEBUG 19:14:34 INFO 19:14:34 INFO [loop_until]: kubectl --namespace=xlou top node 19:14:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:14:34 INFO [loop_until]: OK (rc = 0) 19:14:34 DEBUG --- stdout --- 19:14:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5091Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10754m 36% 10670Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13163m 44% 11109Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17628m 59% 3052Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2090m 7% 7111Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16716m 55% 12512Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2971Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12091m 40% 10543Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1995m 6% 6256Mi 5% gke-xlou-cdm-ds-0f601751-j488 10653m 35% 10115Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2443Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17199m 57% 10484Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13059m 43% 7090Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6890m 43% 5320Mi 9% 19:14:34 DEBUG --- stderr --- 19:14:34 DEBUG 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204933" 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "ds-idrepo-1" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "ds-cts-2" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "0.00009999999999999998" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "instance": "10.142.0.107:9100" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "instance": "10.142.0.43:9100" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "0.0006666666666666673" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204933" 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204933" 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204933" 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "am-7d597cf7d6-8vr8f" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "0" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 ERROR [http_cmd]: ERROR 19:15:33 ERROR --- status code --- 19:15:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:15:33 ERROR --- http response --- 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:15:33 DEBUG --- status code --- 19:15:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "instance": "10.142.0.107:9100" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "instance": "10.142.0.43:9100" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "5470.133333333334" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "ds-idrepo-1" 19:15:33 DEBUG }, 19:15:33 DEBUG "instance": "10.142.0.107:9100" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:15:33 DEBUG 19:15:33 DEBUG "pod": "ds-idrepo-0" 19:15:33 DEBUG }, 19:15:33 DEBUG "instance": "10.142.0.43:9100" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "2301.0333333333333" 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "0" 19:15:33 DEBUG ] 19:15:33 DEBUG "3084.5666666666666" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204933" 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "0" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG { 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:15:33 DEBUG "data": { 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG { 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:15:33 DEBUG "result": [ 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "data": { 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:15:33 DEBUG "result": [ 19:15:33 DEBUG "data": { 19:15:33 DEBUG }, 19:15:33 DEBUG { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "value": [ 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "result": [ 19:15:33 DEBUG 19:15:33 DEBUG "instance": "10.142.0.107:9100" 19:15:33 DEBUG { 19:15:33 DEBUG 19:15:33 DEBUG }, 19:15:33 DEBUG "metric": { 19:15:33 DEBUG 19:15:33 DEBUG "value": [ 19:15:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG }, 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "value": [ 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG }, 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG "value": [ 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG "instance": "10.142.0.43:9100" 19:15:33 DEBUG 19:15:33 DEBUG "2300.2666666666664" 19:15:33 DEBUG { 19:15:33 DEBUG }, 19:15:33 DEBUG 19:15:33 DEBUG ] 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "value": [ 19:15:33 DEBUG "pod": "am-7d597cf7d6-8vr8f" 19:15:33 DEBUG } 19:15:33 DEBUG "data": { 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG }, 19:15:33 DEBUG ] 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "13782635520" 19:15:33 DEBUG "value": [ 19:15:33 DEBUG } 19:15:33 DEBUG "result": [ 19:15:33 DEBUG ] 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG } 19:15:33 DEBUG { 19:15:33 DEBUG } 19:15:33 DEBUG "6.3579547018822975" 19:15:33 DEBUG "metric": { 19:15:33 DEBUG ] 19:15:33 DEBUG ] 19:15:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 DEBUG }, 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG "value": [ 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204933" 19:15:33 DEBUG 19:15:33 DEBUG } 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "2300.9333333333334" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204933" 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "ds-idrepo-1" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "ds-idrepo-0" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "0" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "instance": "10.142.0.107:9100" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "instance": "10.142.0.43:9100" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "0.00033333333333350387" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "6.760738115879082" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "0.078" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "ds-idrepo-1" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "ds-idrepo-0" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "0" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "ds-idrepo-1" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "ds-idrepo-0" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "0" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204933" 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "1655431168" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "ds-idrepo-1" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "ds-cts-2" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "0" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "ds-idrepo-1" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "ds-idrepo-0" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "0" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "am-7d597cf7d6-8vr8f" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "259142625.04545683" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "am-7d597cf7d6-8vr8f" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "3739528.2798892525" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "413389" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204933" 19:15:33 INFO [http_cmd]: http status code OK 19:15:33 DEBUG --- status code --- 19:15:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:15:33 DEBUG --- http response --- 19:15:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:15:33 DEBUG ----- output ----- 19:15:33 DEBUG { 19:15:33 DEBUG "status": "success", 19:15:33 DEBUG "data": { 19:15:33 DEBUG "resultType": "vector", 19:15:33 DEBUG "result": [ 19:15:33 DEBUG { 19:15:33 DEBUG "metric": { 19:15:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG [...] 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG 19:15:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:15:33 DEBUG }, 19:15:33 DEBUG "value": [ 19:15:33 DEBUG 1713204933, 19:15:33 DEBUG "2301.1" 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG ] 19:15:33 DEBUG } 19:15:33 DEBUG } 19:15:34 INFO 19:15:34 INFO [loop_until]: kubectl --namespace=xlou top pods 19:15:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:15:34 INFO [loop_until]: OK (rc = 0) 19:15:34 DEBUG --- stdout --- 19:15:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 12020m 9007Mi am-7d597cf7d6-8vr8f 11681m 9008Mi am-7d597cf7d6-9wz9h 15799m 8829Mi ds-cts-0 15146m 8191Mi ds-cts-1 10824m 8438Mi ds-cts-2 10619m 8574Mi ds-idrepo-0 12402m 4942Mi ds-idrepo-1 1918m 4340Mi ds-idrepo-2 1897m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1022Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6738m 1487Mi 19:15:34 DEBUG --- stderr --- 19:15:34 DEBUG 19:15:34 INFO 19:15:34 INFO [loop_until]: kubectl --namespace=xlou top node 19:15:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:15:34 INFO [loop_until]: OK (rc = 0) 19:15:34 DEBUG --- stdout --- 19:15:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5107Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10833m 36% 10669Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 12633m 42% 11098Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 17316m 58% 3057Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1959m 6% 7126Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16632m 55% 12500Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2967Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 12505m 41% 10536Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2019m 6% 6262Mi 5% gke-xlou-cdm-ds-0f601751-j488 10602m 35% 10122Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2444Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 17105m 57% 10483Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 13085m 43% 7097Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 6858m 43% 4553Mi 7% 19:15:34 DEBUG --- stderr --- 19:15:34 DEBUG 19:15:44 WARNING File /tmp/lodemon_data-20240415_184321/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [15/Apr/2024 19:15:44] "GET /monitoring/average?start_time=20240415_174634&stop_time=20240415_181344 HTTP/1.1" 200 - 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG { 19:16:33 DEBUG { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "data": { 19:16:33 DEBUG "data": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "pod": "ds-idrepo-1" 19:16:33 DEBUG "instance": "10.142.0.107:9100" 19:16:33 DEBUG }, 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "pod": "ds-cts-2" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204993" 19:16:33 DEBUG "instance": "10.142.0.43:9100" 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG }, 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG }, 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204993" 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG "value": [ 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG "value": [ 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG 1713204993, 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG "0.000033333333333333335" 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG "3073.7999999999997" 19:16:33 DEBUG ] 19:16:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:16:33 DEBUG ] 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:33 DEBUG } 19:16:33 DEBUG --- http response --- 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG } 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG ] 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:16:33 DEBUG { 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG ] 19:16:33 DEBUG { 19:16:33 DEBUG } 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG } 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG } 19:16:33 DEBUG { 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG { 19:16:33 DEBUG "data": { 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG } 19:16:33 DEBUG "data": { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "data": { 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG "data": { 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713204993" 19:16:33 DEBUG "result": [ 19:16:33 DEBUG "result": [ 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG { 19:16:33 DEBUG { 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG "result": [ 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG { 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713204993" 19:16:33 DEBUG { 19:16:33 DEBUG "data": { 19:16:33 DEBUG { 19:16:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:16:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "metric": { 19:16:33 DEBUG }, 19:16:33 DEBUG }, 19:16:33 DEBUG "data": { 19:16:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:16:33 DEBUG "result": [ 19:16:33 DEBUG "pod": "ds-idrepo-1" 19:16:33 DEBUG "value": [ 19:16:33 DEBUG "value": [ 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG }, 19:16:33 DEBUG { 19:16:33 DEBUG }, 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "result": [ 19:16:33 DEBUG "value": [ 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG { 19:16:33 DEBUG 19:16:33 DEBUG "instance": "10.142.0.107:9100" 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "metric": { 19:16:33 DEBUG 19:16:33 DEBUG }, 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG [...] 19:16:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:16:33 DEBUG 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG }, 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG 19:16:33 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:16:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG --- http response --- 19:16:33 ERROR [http_cmd]: ERROR 19:16:33 DEBUG 19:16:33 DEBUG }, 19:16:33 DEBUG }, 19:16:33 DEBUG 19:16:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:16:33 DEBUG 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:33 ERROR --- status code --- 19:16:33 DEBUG "pod": "ds-idrepo-0" 19:16:33 DEBUG "value": [ 19:16:33 DEBUG "value": [ 19:16:33 DEBUG [...] 19:16:33 DEBUG }, 19:16:33 DEBUG 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG ----- output ----- 19:16:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:16:33 DEBUG }, 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG { 19:16:33 ERROR --- http response --- 19:16:33 DEBUG "value": [ 19:16:33 DEBUG "0" 19:16:33 DEBUG "2294.3333333333335" 19:16:33 DEBUG 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "instance": "10.142.0.43:9100" 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:16:33 DEBUG ] 19:16:33 DEBUG ] 19:16:33 DEBUG 19:16:33 DEBUG "252.6" 19:16:33 DEBUG }, 19:16:33 DEBUG "data": { 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:33 DEBUG "0" 19:16:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:16:33 DEBUG ] 19:16:33 DEBUG "value": [ 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG ] 19:16:33 DEBUG ] 19:16:33 DEBUG ] 19:16:33 DEBUG }, 19:16:33 DEBUG } 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 DEBUG "value": [ 19:16:33 DEBUG ] 19:16:33 DEBUG "0.00033333333333333365" 19:16:33 DEBUG { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "data": { 19:16:33 DEBUG } 19:16:33 DEBUG "0" 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 DEBUG "instance": "10.142.0.107:9100" 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG ] 19:16:33 DEBUG }, 19:16:33 DEBUG "result": [ 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 DEBUG "value": [ 19:16:33 DEBUG { 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG 19:16:33 DEBUG "metric": { 19:16:33 DEBUG } 19:16:33 DEBUG 19:16:33 DEBUG "instance": "10.142.0.107:9100" 19:16:33 DEBUG } 19:16:33 DEBUG 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG }, 19:16:33 DEBUG [...] 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:33 DEBUG "instance": "10.142.0.43:9100" 19:16:33 DEBUG [...] 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG }, 19:16:33 DEBUG 19:16:33 DEBUG { 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG 19:16:33 DEBUG "data": { 19:16:33 DEBUG "0.0006333333333335152" 19:16:33 DEBUG "instance": "10.142.0.43:9100" 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG ] 19:16:33 DEBUG }, 19:16:33 DEBUG "result": [ 19:16:33 DEBUG } 19:16:33 DEBUG "value": [ 19:16:33 DEBUG { 19:16:33 DEBUG ] 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "metric": { 19:16:33 DEBUG } 19:16:33 DEBUG "13783305216" 19:16:33 DEBUG "instance": "10.142.0.107:9100" 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG }, 19:16:33 DEBUG } 19:16:33 DEBUG "value": [ 19:16:33 DEBUG ] 19:16:33 DEBUG 19:16:33 DEBUG } 19:16:33 DEBUG 19:16:33 DEBUG } 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "instance": "10.142.0.43:9100" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "5381.366666666667" 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713204993" 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "data": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "0" 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG { 19:16:33 DEBUG { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "data": { 19:16:33 DEBUG "data": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "pod": "ds-idrepo-1" 19:16:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:16:33 DEBUG }, 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "pod": "ds-idrepo-0" 19:16:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:16:33 DEBUG }, 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "0" 19:16:33 DEBUG "252.2" 19:16:33 DEBUG ] 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "data": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "1.9990209420373235" 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "data": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "pod": "ds-idrepo-1" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "pod": "ds-idrepo-0" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "0" 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "data": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "0.078" 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "data": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "pod": "ds-idrepo-1" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "pod": "ds-idrepo-0" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "0" 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713204993" 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "data": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "872570880" 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "data": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "pod": "ds-idrepo-1" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "pod": "ds-cts-2" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "0" 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "data": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "pod": "ds-idrepo-1" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "pod": "ds-idrepo-0" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "0" 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG { 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG "data": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "327970.5" 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "data": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "0" 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "data": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "0" 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713204993" 19:16:33 INFO [http_cmd]: http status code OK 19:16:33 DEBUG --- status code --- 19:16:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:16:33 DEBUG --- http response --- 19:16:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:16:33 DEBUG ----- output ----- 19:16:33 DEBUG { 19:16:33 DEBUG "status": "success", 19:16:33 DEBUG "data": { 19:16:33 DEBUG "resultType": "vector", 19:16:33 DEBUG "result": [ 19:16:33 DEBUG { 19:16:33 DEBUG "metric": { 19:16:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG [...] 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG 19:16:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:16:33 DEBUG }, 19:16:33 DEBUG "value": [ 19:16:33 DEBUG 1713204993, 19:16:33 DEBUG "252.7" 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG ] 19:16:33 DEBUG } 19:16:33 DEBUG } 19:16:34 INFO 19:16:34 INFO [loop_until]: kubectl --namespace=xlou top pods 19:16:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:34 INFO [loop_until]: OK (rc = 0) 19:16:34 DEBUG --- stdout --- 19:16:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 3100m 9010Mi am-7d597cf7d6-8vr8f 3867m 9011Mi am-7d597cf7d6-9wz9h 3845m 8826Mi ds-cts-0 8853m 8152Mi ds-cts-1 5387m 8408Mi ds-cts-2 5786m 8583Mi ds-idrepo-0 2894m 4941Mi ds-idrepo-1 282m 4340Mi ds-idrepo-2 329m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 3416m 1150Mi 19:16:34 DEBUG --- stderr --- 19:16:34 DEBUG 19:16:34 INFO 19:16:34 INFO [loop_until]: kubectl --namespace=xlou top node 19:16:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:16:34 INFO [loop_until]: OK (rc = 0) 19:16:34 DEBUG --- stdout --- 19:16:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 5108Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 6497m 21% 10658Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 4928m 16% 11110Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 3723m 12% 3058Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 733m 2% 7125Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 9178m 30% 12512Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2964Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 4161m 13% 10549Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 525m 1% 6256Mi 5% gke-xlou-cdm-ds-0f601751-j488 5661m 18% 10118Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2443Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 4138m 13% 10510Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 2753m 9% 7104Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 3538m 22% 4983Mi 8% 19:16:34 DEBUG --- stderr --- 19:16:34 DEBUG 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205053" 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "2323" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205053" 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205053" 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205053" 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "pod": "ds-idrepo-1" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "ds-cts-2" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "0.0003862600000000001" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713205053" 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:33 DEBUG --- http response --- 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG { 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG "data": { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG "result": [ 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG { 19:17:33 DEBUG "result": [ 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:33 DEBUG { 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "instance": "10.142.0.107:9100" 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "instance": "10.142.0.107:9100" 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205053" 19:17:33 DEBUG }, 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "instance": "10.142.0.107:9100" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG "instance": "10.142.0.107:9100" 19:17:33 DEBUG "data": { 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG }, 19:17:33 ERROR [http_cmd]: ERROR 19:17:33 DEBUG "value": [ 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG 19:17:33 DEBUG }, 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG "value": [ 19:17:33 ERROR --- status code --- 19:17:33 DEBUG 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG 19:17:33 DEBUG "value": [ 19:17:33 DEBUG "result": [ 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG 19:17:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:17:33 DEBUG 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG { 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG 19:17:33 ERROR --- http response --- 19:17:33 DEBUG 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG "metric": { 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:33 DEBUG 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG [...] 19:17:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG [...] 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG 19:17:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG }, 19:17:33 DEBUG { 19:17:33 DEBUG 19:17:33 DEBUG { 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG 19:17:33 DEBUG { 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "value": [ 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "instance": "10.142.0.43:9100" 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "data": { 19:17:33 DEBUG 19:17:33 DEBUG "data": { 19:17:33 DEBUG "instance": "10.142.0.43:9100" 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:17:33 DEBUG "data": { 19:17:33 DEBUG }, 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "resultType": "vector", 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 DEBUG "instance": "10.142.0.43:9100" 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG }, 19:17:33 DEBUG ----- output ----- 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "value": [ 19:17:33 DEBUG "instance": "10.142.0.43:9100" 19:17:33 DEBUG 19:17:33 DEBUG "result": [ 19:17:33 DEBUG }, 19:17:33 DEBUG "result": [ 19:17:33 DEBUG "value": [ 19:17:33 DEBUG { 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG "result": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG }, 19:17:33 DEBUG [...] 19:17:33 DEBUG { 19:17:33 DEBUG "value": [ 19:17:33 DEBUG { 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG { 19:17:33 DEBUG "6187.633333333333" 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG "metric": { 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "13784009728" 19:17:33 DEBUG "data": { 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG "metric": { 19:17:33 DEBUG ] 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG 19:17:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:17:33 DEBUG "0.00030000000000001136" 19:17:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:17:33 DEBUG ] 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:33 DEBUG } 19:17:33 DEBUG "0" 19:17:33 DEBUG 19:17:33 DEBUG }, 19:17:33 DEBUG ] 19:17:33 DEBUG }, 19:17:33 DEBUG } 19:17:33 DEBUG "result": [ 19:17:33 DEBUG }, 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG ] 19:17:33 DEBUG ] 19:17:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:17:33 DEBUG "value": [ 19:17:33 DEBUG } 19:17:33 DEBUG "value": [ 19:17:33 DEBUG ] 19:17:33 DEBUG { 19:17:33 DEBUG "value": [ 19:17:33 DEBUG { 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 DEBUG }, 19:17:33 DEBUG 19:17:33 DEBUG ] 19:17:33 DEBUG 19:17:33 DEBUG } 19:17:33 DEBUG "metric": { 19:17:33 DEBUG 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG } 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "ds-idrepo-1" 19:17:33 DEBUG 19:17:33 DEBUG "data": { 19:17:33 DEBUG } 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 DEBUG [...] 19:17:33 DEBUG }, 19:17:33 DEBUG 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG } 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG "value": [ 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG { 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "0" 19:17:33 DEBUG 19:17:33 DEBUG "metric": { 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG ] 19:17:33 DEBUG 19:17:33 DEBUG "instance": "10.142.0.107:9100" 19:17:33 DEBUG 19:17:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:17:33 DEBUG 19:17:33 DEBUG } 19:17:33 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:17:33 DEBUG }, 19:17:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:17:33 DEBUG }, 19:17:33 DEBUG [...] 19:17:33 DEBUG ] 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG } 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG 19:17:33 DEBUG } 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "2485.1" 19:17:33 DEBUG 19:17:33 DEBUG "0" 19:17:33 DEBUG 19:17:33 DEBUG "2484.6" 19:17:33 DEBUG ] 19:17:33 DEBUG "pod": "ds-idrepo-0" 19:17:33 DEBUG ] 19:17:33 DEBUG [...] 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG }, 19:17:33 DEBUG } 19:17:33 DEBUG 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG "value": [ 19:17:33 DEBUG ] 19:17:33 DEBUG 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG } 19:17:33 DEBUG 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 DEBUG "0" 19:17:33 DEBUG } 19:17:33 DEBUG "instance": "10.142.0.43:9100" 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG }, 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG } 19:17:33 DEBUG "value": [ 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:33 DEBUG ] 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG } 19:17:33 DEBUG "6101.6" 19:17:33 DEBUG { 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG } 19:17:33 DEBUG "data": { 19:17:33 DEBUG ] 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG } 19:17:33 DEBUG "result": [ 19:17:33 DEBUG } 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "0.078" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "pod": "ds-idrepo-1" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "ds-idrepo-0" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "0" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713205053" 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "pod": "ds-idrepo-1" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "ds-idrepo-0" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "0" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "0.001360745732798946" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "7.039344027327875" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "pod": "ds-idrepo-1" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "ds-idrepo-0" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "0" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713205053" 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "1510899712" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "293469.5" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "pod": "ds-idrepo-1" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "ds-cts-2" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "0" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "pod": "ds-idrepo-1" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "ds-idrepo-0" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "0" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "37123075.59359297" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "0" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205053" 19:17:33 INFO [http_cmd]: http status code OK 19:17:33 DEBUG --- status code --- 19:17:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:17:33 DEBUG --- http response --- 19:17:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:17:33 DEBUG ----- output ----- 19:17:33 DEBUG { 19:17:33 DEBUG "status": "success", 19:17:33 DEBUG "data": { 19:17:33 DEBUG "resultType": "vector", 19:17:33 DEBUG "result": [ 19:17:33 DEBUG { 19:17:33 DEBUG "metric": { 19:17:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG [...] 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG 19:17:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:17:33 DEBUG }, 19:17:33 DEBUG "value": [ 19:17:33 DEBUG 1713205053, 19:17:33 DEBUG "2486.7" 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG ] 19:17:33 DEBUG } 19:17:33 DEBUG } 19:17:34 INFO 19:17:34 INFO [loop_until]: kubectl --namespace=xlou top pods 19:17:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:17:34 INFO [loop_until]: OK (rc = 0) 19:17:34 DEBUG --- stdout --- 19:17:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 14008m 9066Mi am-7d597cf7d6-8vr8f 12264m 9011Mi am-7d597cf7d6-9wz9h 17903m 8865Mi ds-cts-0 15629m 8199Mi ds-cts-1 11509m 8456Mi ds-cts-2 11036m 8602Mi ds-idrepo-0 13922m 4941Mi ds-idrepo-1 2155m 4341Mi ds-idrepo-2 2087m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7776m 1412Mi 19:17:34 DEBUG --- stderr --- 19:17:34 DEBUG 19:17:35 INFO 19:17:35 INFO [loop_until]: kubectl --namespace=xlou top node 19:17:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:17:35 INFO [loop_until]: OK (rc = 0) 19:17:35 DEBUG --- stdout --- 19:17:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 5104Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 10753m 36% 10690Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 14262m 47% 11154Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 20097m 67% 3151Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2248m 7% 7117Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16887m 56% 12495Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2978Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 13465m 45% 10537Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2203m 7% 6252Mi 5% gke-xlou-cdm-ds-0f601751-j488 11059m 37% 10131Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2435Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 18910m 63% 10513Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14047m 47% 7092Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 7594m 47% 5404Mi 9% 19:17:35 DEBUG --- stderr --- 19:17:35 DEBUG 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "2304.5" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205113" 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "instance": "10.142.0.107:9100" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "instance": "10.142.0.43:9100" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "6917.133333333333" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205113" 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "2465.2666666666664" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "0.078" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713205113" 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "instance": "10.142.0.107:9100" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "instance": "10.142.0.43:9100" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "0.0005333333333335114" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "ds-idrepo-1" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "ds-cts-2" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "0.00003333333333333336" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "ds-idrepo-1" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "ds-idrepo-0" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "0" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205113" 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205113" 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205113" 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 ERROR [http_cmd]: ERROR 19:18:33 ERROR --- status code --- 19:18:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:18:33 ERROR --- http response --- 19:18:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:18:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713205113" 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "2465.7" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:33 DEBUG "result": [ 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:33 DEBUG "result": [ 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "instance": "10.142.0.107:9100" 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "instance": "10.142.0.107:9100" 19:18:33 DEBUG "data": { 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 DEBUG }, 19:18:33 DEBUG "data": { 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG }, 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "value": [ 19:18:33 DEBUG "resultType": "vector", 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG "value": [ 19:18:33 DEBUG "result": [ 19:18:33 DEBUG 19:18:33 DEBUG "result": [ 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG 19:18:33 DEBUG { 19:18:33 DEBUG 19:18:33 DEBUG { 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG 19:18:33 DEBUG "metric": { 19:18:33 DEBUG 19:18:33 DEBUG "metric": { 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:18:33 DEBUG [...] 19:18:33 DEBUG "instance": "10.142.0.107:9100" 19:18:33 DEBUG [...] 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG }, 19:18:33 DEBUG 19:18:33 DEBUG }, 19:18:33 DEBUG 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG 19:18:33 DEBUG "instance": "10.142.0.43:9100" 19:18:33 DEBUG 19:18:33 DEBUG "instance": "10.142.0.43:9100" 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG 19:18:33 DEBUG }, 19:18:33 DEBUG 19:18:33 DEBUG }, 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG [...] 19:18:33 DEBUG "value": [ 19:18:33 DEBUG [...] 19:18:33 DEBUG "value": [ 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG 19:18:33 DEBUG "13784704000" 19:18:33 DEBUG 19:18:33 DEBUG "6307.766666666667" 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "ds-cts-1" 19:18:33 DEBUG 19:18:33 DEBUG ] 19:18:33 DEBUG 19:18:33 DEBUG ] 19:18:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:18:33 DEBUG }, 19:18:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:18:33 DEBUG } 19:18:33 DEBUG "instance": "10.142.0.43:9100" 19:18:33 DEBUG } 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG }, 19:18:33 DEBUG ] 19:18:33 DEBUG }, 19:18:33 DEBUG ] 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG "value": [ 19:18:33 DEBUG } 19:18:33 DEBUG "value": [ 19:18:33 DEBUG } 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG } 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG } 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "0" 19:18:33 DEBUG "0" 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG ] 19:18:33 DEBUG ] 19:18:33 DEBUG [...] 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 DEBUG 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG 19:18:33 DEBUG ] 19:18:33 DEBUG ] 19:18:33 DEBUG 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 DEBUG 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:33 DEBUG }, 19:18:33 DEBUG }, 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG "value": [ 19:18:33 DEBUG "value": [ 19:18:33 DEBUG { 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "0" 19:18:33 DEBUG "7.883466903072156" 19:18:33 DEBUG "data": { 19:18:33 DEBUG ] 19:18:33 DEBUG ] 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 DEBUG "result": [ 19:18:33 DEBUG ] 19:18:33 DEBUG ] 19:18:33 DEBUG { 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 DEBUG "metric": { 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 DEBUG "pod": "ds-idrepo-1" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "ds-idrepo-0" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "0" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "ds-idrepo-1" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "ds-idrepo-0" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "0" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "ds-cts-1" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "0.000013299336986671241" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "ds-idrepo-1" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "ds-idrepo-0" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "0" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713205113" 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "1880076288" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "302483" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "ds-cts-1" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "3592.9715174886674" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "ds-idrepo-1" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "ds-cts-2" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "0" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "ds-idrepo-1" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "ds-idrepo-0" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "0" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "ds-cts-1" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "0" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205113" 19:18:33 INFO [http_cmd]: http status code OK 19:18:33 DEBUG --- status code --- 19:18:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:18:33 DEBUG --- http response --- 19:18:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:18:33 DEBUG ----- output ----- 19:18:33 DEBUG { 19:18:33 DEBUG "status": "success", 19:18:33 DEBUG "data": { 19:18:33 DEBUG "resultType": "vector", 19:18:33 DEBUG "result": [ 19:18:33 DEBUG { 19:18:33 DEBUG "metric": { 19:18:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG [...] 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG 19:18:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:18:33 DEBUG }, 19:18:33 DEBUG "value": [ 19:18:33 DEBUG 1713205113, 19:18:33 DEBUG "2465.766666666667" 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG ] 19:18:33 DEBUG } 19:18:33 DEBUG } 19:18:34 INFO 19:18:34 INFO [loop_until]: kubectl --namespace=xlou top pods 19:18:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:18:34 INFO [loop_until]: OK (rc = 0) 19:18:34 DEBUG --- stdout --- 19:18:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 13838m 9018Mi am-7d597cf7d6-8vr8f 12640m 9013Mi am-7d597cf7d6-9wz9h 17221m 9221Mi ds-cts-0 15380m 8221Mi ds-cts-1 11024m 8432Mi ds-cts-2 11051m 8591Mi ds-idrepo-0 13842m 4941Mi ds-idrepo-1 2127m 4340Mi ds-idrepo-2 2035m 4112Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1022Mi idm-b85f9d978-rl76f 4m 1222Mi lodemon-7c47b75958-dgfm9 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7875m 2563Mi 19:18:34 DEBUG --- stderr --- 19:18:34 DEBUG 19:18:35 INFO 19:18:35 INFO [loop_until]: kubectl --namespace=xlou top node 19:18:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:18:35 INFO [loop_until]: OK (rc = 0) 19:18:35 DEBUG --- stdout --- 19:18:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5099Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11196m 37% 10678Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 13999m 46% 11114Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 21166m 70% 3259Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2256m 7% 7124Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 16898m 56% 12492Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 68m 0% 2989Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 13628m 45% 10538Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2211m 7% 6266Mi 5% gke-xlou-cdm-ds-0f601751-j488 11003m 36% 10136Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2448Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 18832m 63% 10900Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14119m 47% 7092Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8097m 50% 6493Mi 11% 19:18:35 DEBUG --- stderr --- 19:18:35 DEBUG 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "250.16666666666666" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 ERROR [http_cmd]: ERROR 19:19:33 ERROR --- status code --- 19:19:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:19:33 ERROR --- http response --- 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG --- status code --- 19:19:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG --- http response --- 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG { 19:19:33 DEBUG { 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG "data": { 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "instance": "10.142.0.107:9100" 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:19:33 DEBUG }, 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "pod": "ds-idrepo-1" 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "instance": "10.142.0.107:9100" 19:19:33 DEBUG }, 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "instance": "10.142.0.43:9100" 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:19:33 DEBUG }, 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "5272" 19:19:33 DEBUG 19:19:33 DEBUG "pod": "ds-cts-2" 19:19:33 DEBUG "2482.133333333333" 19:19:33 DEBUG ] 19:19:33 DEBUG "instance": "10.142.0.43:9100" 19:19:33 DEBUG }, 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "0.00030000000000001136" 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 DEBUG "0.00005311333333333334" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "pod": "ds-idrepo-1" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "pod": "ds-idrepo-0" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "0" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "0.078" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205173" 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "7.8538317380284015" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "instance": "10.142.0.107:9100" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "instance": "10.142.0.43:9100" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "13785447424" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "pod": "ds-cts-1" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "0" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "2481.0666666666666" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "0" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG --- status code --- 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG { 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG "data": { 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:33 DEBUG { 19:19:33 DEBUG { 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG "result": [ 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG { 19:19:33 DEBUG { 19:19:33 DEBUG { 19:19:33 DEBUG "data": { 19:19:33 DEBUG "data": { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "instance": "10.142.0.107:9100" 19:19:33 DEBUG "data": { 19:19:33 DEBUG "result": [ 19:19:33 DEBUG "result": [ 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG }, 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG { 19:19:33 DEBUG { 19:19:33 DEBUG "result": [ 19:19:33 DEBUG "value": [ 19:19:33 DEBUG "result": [ 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG { 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG 19:19:33 DEBUG { 19:19:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:19:33 DEBUG "pod": "ds-idrepo-1" 19:19:33 DEBUG "metric": { 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG 19:19:33 DEBUG "metric": { 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG }, 19:19:33 DEBUG }, 19:19:33 DEBUG "instance": "10.142.0.107:9100" 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG 19:19:33 DEBUG "pod": "ds-idrepo-1" 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG "value": [ 19:19:33 DEBUG "value": [ 19:19:33 DEBUG }, 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG [...] 19:19:33 DEBUG }, 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:33 DEBUG "value": [ 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG { 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG "instance": "10.142.0.43:9100" 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG { 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG }, 19:19:33 DEBUG "data": { 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "value": [ 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "data": { 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "result": [ 19:19:33 DEBUG 19:19:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "pod": "ds-idrepo-0" 19:19:33 DEBUG 19:19:33 DEBUG "6426.333333333334" 19:19:33 DEBUG { 19:19:33 DEBUG 19:19:33 DEBUG }, 19:19:33 DEBUG "result": [ 19:19:33 DEBUG }, 19:19:33 DEBUG "instance": "10.142.0.43:9100" 19:19:33 DEBUG ] 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "pod": "ds-idrepo-0" 19:19:33 DEBUG "value": [ 19:19:33 DEBUG { 19:19:33 DEBUG "value": [ 19:19:33 DEBUG }, 19:19:33 DEBUG } 19:19:33 DEBUG "pod": "ds-idrepo-1" 19:19:33 DEBUG }, 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "metric": { 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG ] 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG "2362953728" 19:19:33 DEBUG "pod": "ds-cts-1" 19:19:33 DEBUG "0" 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG } 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG ] 19:19:33 DEBUG }, 19:19:33 DEBUG ] 19:19:33 DEBUG "0.00033333333333333365" 19:19:33 DEBUG } 19:19:33 DEBUG 19:19:33 DEBUG "0" 19:19:33 DEBUG } 19:19:33 DEBUG "value": [ 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG 19:19:33 DEBUG ] 19:19:33 DEBUG ] 19:19:33 DEBUG 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 DEBUG 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG [...] 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 DEBUG 19:19:33 DEBUG } 19:19:33 DEBUG [...] 19:19:33 DEBUG } 19:19:33 DEBUG 19:19:33 DEBUG } 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "pod": "ds-idrepo-0" 19:19:33 DEBUG 19:19:33 DEBUG }, 19:19:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:19:33 DEBUG "value": [ 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "0" 19:19:33 DEBUG "13.825349430984701" 19:19:33 DEBUG ] 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "pod": "ds-cts-1" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "314741945.1527701" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "401541" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "pod": "ds-idrepo-1" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "pod": "ds-cts-2" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "0" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "pod": "ds-idrepo-1" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "pod": "ds-idrepo-0" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "0" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "pod": "ds-cts-1" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "4982195.5848306" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205173" 19:19:33 INFO [http_cmd]: http status code OK 19:19:33 DEBUG --- status code --- 19:19:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:19:33 DEBUG --- http response --- 19:19:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:19:33 DEBUG ----- output ----- 19:19:33 DEBUG { 19:19:33 DEBUG "status": "success", 19:19:33 DEBUG "data": { 19:19:33 DEBUG "resultType": "vector", 19:19:33 DEBUG "result": [ 19:19:33 DEBUG { 19:19:33 DEBUG "metric": { 19:19:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG [...] 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG 19:19:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:19:33 DEBUG }, 19:19:33 DEBUG "value": [ 19:19:33 DEBUG 1713205173, 19:19:33 DEBUG "2497" 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG ] 19:19:33 DEBUG } 19:19:33 DEBUG } 19:19:34 INFO 19:19:34 INFO [loop_until]: kubectl --namespace=xlou top pods 19:19:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:19:34 INFO [loop_until]: OK (rc = 0) 19:19:34 DEBUG --- stdout --- 19:19:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 11390m 9008Mi am-7d597cf7d6-8vr8f 10564m 9013Mi am-7d597cf7d6-9wz9h 14821m 9183Mi ds-cts-0 7882m 8224Mi ds-cts-1 6783m 8462Mi ds-cts-2 6148m 8610Mi ds-idrepo-0 12577m 4941Mi ds-idrepo-1 2030m 4340Mi ds-idrepo-2 1891m 4112Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7808m 2163Mi 19:19:34 DEBUG --- stderr --- 19:19:34 DEBUG 19:19:35 INFO 19:19:35 INFO [loop_until]: kubectl --namespace=xlou top node 19:19:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:19:35 INFO [loop_until]: OK (rc = 0) 19:19:35 DEBUG --- stdout --- 19:19:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5082Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 6650m 22% 10727Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 11875m 39% 11095Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 20012m 67% 3058Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2172m 7% 7126Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 9810m 32% 12522Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2993Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 11596m 38% 10393Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1997m 6% 6270Mi 5% gke-xlou-cdm-ds-0f601751-j488 5935m 19% 10185Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2454Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 15435m 51% 10844Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12782m 42% 7100Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8093m 50% 5988Mi 10% 19:19:35 DEBUG --- stderr --- 19:19:35 DEBUG 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 ERROR [http_cmd]: ERROR 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 ERROR --- status code --- 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG --- status code --- 19:20:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 ERROR --- http response --- 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:20:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "2516.9333333333334" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "instance": "10.142.0.107:9100" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "instance": "10.142.0.43:9100" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "0.00023333333333302637" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG --- http response --- 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG { 19:20:33 DEBUG { 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:20:33 DEBUG "data": { 19:20:33 DEBUG "data": { 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG { 19:20:33 DEBUG "result": [ 19:20:33 DEBUG "result": [ 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG { 19:20:33 DEBUG { 19:20:33 DEBUG "data": { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "pod": "ds-idrepo-1" 19:20:33 DEBUG "instance": "10.142.0.107:9100" 19:20:33 DEBUG "result": [ 19:20:33 DEBUG }, 19:20:33 DEBUG }, 19:20:33 DEBUG { 19:20:33 DEBUG "value": [ 19:20:33 DEBUG "value": [ 19:20:33 DEBUG "metric": { 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG }, 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "value": [ 19:20:33 DEBUG [...] 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG "pod": "ds-idrepo-0" 19:20:33 DEBUG "instance": "10.142.0.43:9100" 19:20:33 DEBUG 19:20:33 DEBUG }, 19:20:33 DEBUG }, 19:20:33 DEBUG 19:20:33 DEBUG "value": [ 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:20:33 DEBUG "0" 19:20:33 DEBUG "4733.633333333333" 19:20:33 DEBUG }, 19:20:33 DEBUG ] 19:20:33 DEBUG ] 19:20:33 DEBUG "value": [ 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG ] 19:20:33 DEBUG ] 19:20:33 DEBUG "7.959377464394988" 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG } 19:20:33 DEBUG --- status code --- 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG } 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:20:33 DEBUG { 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG { 19:20:33 DEBUG "data": { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "result": [ 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG { 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "result": [ 19:20:33 DEBUG "metric": { 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG { 19:20:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:20:33 DEBUG "metric": { 19:20:33 DEBUG }, 19:20:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:20:33 DEBUG }, 19:20:33 DEBUG "pod": "ds-idrepo-1" 19:20:33 DEBUG "value": [ 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG }, 19:20:33 DEBUG 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:20:33 DEBUG 19:20:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:20:33 DEBUG 19:20:33 DEBUG }, 19:20:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:20:33 DEBUG }, 19:20:33 DEBUG "pod": "ds-cts-2" 19:20:33 DEBUG "value": [ 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG }, 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG "0.078" 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "2485.4" 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG ] 19:20:33 DEBUG "2486" 19:20:33 DEBUG ] 19:20:33 DEBUG "0.000033333333333333335" 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "pod": "ds-idrepo-1" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "pod": "ds-idrepo-0" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "0" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:20:33 DEBUG "instance": "10.142.0.107:9100" 19:20:33 DEBUG }, 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "pod": "ds-cts-0" 19:20:33 DEBUG 19:20:33 DEBUG }, 19:20:33 DEBUG "instance": "10.142.0.43:9100" 19:20:33 DEBUG "value": [ 19:20:33 DEBUG }, 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG "30940867.714391783" 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG ] 19:20:33 DEBUG "13785959424" 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "0" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "instance": "10.142.0.107:9100" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "instance": "10.142.0.43:9100" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "5709.133333333334" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "pod": "ds-cts-0" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "16.760913964856" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:33 DEBUG "metric": { 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG "pod": "ds-idrepo-1" 19:20:33 DEBUG { 19:20:33 DEBUG }, 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "value": [ 19:20:33 DEBUG "data": { 19:20:33 DEBUG 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG 19:20:33 DEBUG "result": [ 19:20:33 DEBUG 19:20:33 DEBUG { 19:20:33 DEBUG [...] 19:20:33 DEBUG "metric": { 19:20:33 DEBUG 19:20:33 DEBUG "pod": "ds-idrepo-1" 19:20:33 DEBUG 19:20:33 DEBUG }, 19:20:33 DEBUG 19:20:33 DEBUG "value": [ 19:20:33 DEBUG "pod": "ds-idrepo-0" 19:20:33 DEBUG 19:20:33 DEBUG }, 19:20:33 DEBUG 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG [...] 19:20:33 DEBUG "0" 19:20:33 DEBUG 19:20:33 DEBUG ] 19:20:33 DEBUG 19:20:33 DEBUG } 19:20:33 DEBUG 19:20:33 DEBUG ] 19:20:33 DEBUG "pod": "ds-idrepo-0" 19:20:33 DEBUG } 19:20:33 DEBUG }, 19:20:33 DEBUG } 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "0" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "2371706880" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "instance": "10.142.0.107:9100" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "instance": "10.142.0.43:9100" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "0.00033333333333333365" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "pod": "ds-cts-0" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "266126715.01051852" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "pod": "ds-idrepo-1" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "pod": "ds-cts-2" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "0" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "445984" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "pod": "ds-cts-0" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "47403685.68246504" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "pod": "ds-idrepo-1" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "pod": "ds-idrepo-0" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "0" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205233" 19:20:33 INFO [http_cmd]: http status code OK 19:20:33 DEBUG --- status code --- 19:20:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:20:33 DEBUG --- http response --- 19:20:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:20:33 DEBUG ----- output ----- 19:20:33 DEBUG { 19:20:33 DEBUG "status": "success", 19:20:33 DEBUG "data": { 19:20:33 DEBUG "resultType": "vector", 19:20:33 DEBUG "result": [ 19:20:33 DEBUG { 19:20:33 DEBUG "metric": { 19:20:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG [...] 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG 19:20:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:20:33 DEBUG }, 19:20:33 DEBUG "value": [ 19:20:33 DEBUG 1713205233, 19:20:33 DEBUG "2485.2" 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG ] 19:20:33 DEBUG } 19:20:33 DEBUG } 19:20:34 INFO 19:20:34 INFO [loop_until]: kubectl --namespace=xlou top pods 19:20:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:20:34 INFO [loop_until]: OK (rc = 0) 19:20:34 DEBUG --- stdout --- 19:20:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 13862m 9010Mi am-7d597cf7d6-8vr8f 12946m 8880Mi am-7d597cf7d6-9wz9h 17568m 9206Mi ds-cts-0 16639m 8401Mi ds-cts-1 11961m 8443Mi ds-cts-2 11575m 8592Mi ds-idrepo-0 13873m 4941Mi ds-idrepo-1 2199m 4341Mi ds-idrepo-2 2096m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7993m 2177Mi 19:20:34 DEBUG --- stderr --- 19:20:34 DEBUG 19:20:35 INFO 19:20:35 INFO [loop_until]: kubectl --namespace=xlou top node 19:20:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:20:35 INFO [loop_until]: OK (rc = 0) 19:20:35 DEBUG --- stdout --- 19:20:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5090Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 12226m 40% 10693Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 14488m 48% 11097Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 21080m 70% 3210Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 2205m 7% 7129Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 17500m 58% 12710Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 2977Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 13958m 46% 10401Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2193m 7% 6274Mi 5% gke-xlou-cdm-ds-0f601751-j488 11726m 39% 10147Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2441Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 18961m 63% 10874Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14173m 47% 7098Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8191m 51% 5988Mi 10% 19:20:35 DEBUG --- stderr --- 19:20:35 DEBUG 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "8.030813266668567" 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "2498.4666666666667" 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG 1713205293, 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG "2532.0666666666666" 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG { 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG ] 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 ERROR [http_cmd]: ERROR 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG } 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:21:33 ERROR --- status code --- 19:21:33 DEBUG "data": { 19:21:33 DEBUG ] 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG } 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG ----- output ----- 19:21:33 ERROR --- http response --- 19:21:33 DEBUG "result": [ 19:21:33 DEBUG } 19:21:33 DEBUG { 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG { 19:21:33 DEBUG { 19:21:33 DEBUG { 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 DEBUG { 19:21:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "metric": { 19:21:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:21:33 DEBUG "data": { 19:21:33 DEBUG "data": { 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:33 DEBUG "data": { 19:21:33 DEBUG "data": { 19:21:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "resultType": "vector", 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG }, 19:21:33 DEBUG "result": [ 19:21:33 DEBUG "result": [ 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG { 19:21:33 DEBUG "result": [ 19:21:33 DEBUG "result": [ 19:21:33 DEBUG "value": [ 19:21:33 DEBUG { 19:21:33 DEBUG { 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG { 19:21:33 DEBUG { 19:21:33 DEBUG 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG "data": { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG 19:21:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:21:33 DEBUG "instance": "10.142.0.107:9100" 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:21:33 DEBUG "pod": "ds-cts-1" 19:21:33 DEBUG "pod": "ds-idrepo-1" 19:21:33 DEBUG 19:21:33 DEBUG }, 19:21:33 DEBUG }, 19:21:33 DEBUG "result": [ 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG }, 19:21:33 DEBUG }, 19:21:33 DEBUG [...] 19:21:33 DEBUG "value": [ 19:21:33 DEBUG "value": [ 19:21:33 DEBUG { 19:21:33 DEBUG { 19:21:33 DEBUG "value": [ 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "instance": "10.142.0.107:9100" 19:21:33 DEBUG "data": { 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG }, 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:21:33 DEBUG [...] 19:21:33 DEBUG [...] 19:21:33 DEBUG "value": [ 19:21:33 DEBUG "result": [ 19:21:33 DEBUG [...] 19:21:33 DEBUG [...] 19:21:33 DEBUG }, 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG { 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "metric": { 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "pod": "ds-idrepo-1" 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "2499.366666666667" 19:21:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:21:33 DEBUG "instance": "10.142.0.43:9100" 19:21:33 DEBUG [...] 19:21:33 DEBUG }, 19:21:33 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 19:21:33 DEBUG "pod": "ds-idrepo-0" 19:21:33 DEBUG ] 19:21:33 DEBUG }, 19:21:33 DEBUG }, 19:21:33 DEBUG 19:21:33 DEBUG "value": [ 19:21:33 DEBUG }, 19:21:33 DEBUG }, 19:21:33 DEBUG } 19:21:33 DEBUG "value": [ 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "value": [ 19:21:33 DEBUG "value": [ 19:21:33 DEBUG ] 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG } 19:21:33 DEBUG "0.078" 19:21:33 DEBUG "0.0003333333333330302" 19:21:33 DEBUG "instance": "10.142.0.43:9100" 19:21:33 DEBUG 19:21:33 DEBUG "0" 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG "0" 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG ] 19:21:33 DEBUG }, 19:21:33 DEBUG [...] 19:21:33 DEBUG ] 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG } 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG ] 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG 19:21:33 DEBUG ] 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 DEBUG "5133.433333333333" 19:21:33 DEBUG 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG "pod": "ds-idrepo-0" 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG } 19:21:33 DEBUG }, 19:21:33 DEBUG { 19:21:33 DEBUG ] 19:21:33 DEBUG "value": [ 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG } 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "data": { 19:21:33 DEBUG } 19:21:33 DEBUG "0" 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG ] 19:21:33 DEBUG "result": [ 19:21:33 DEBUG } 19:21:33 DEBUG { 19:21:33 DEBUG ] 19:21:33 DEBUG "metric": { 19:21:33 DEBUG } 19:21:33 DEBUG "pod": "ds-idrepo-1" 19:21:33 DEBUG } 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "pod": "ds-cts-2" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "0.000019780000000000004" 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205293" 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "instance": "10.142.0.107:9100" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "instance": "10.142.0.43:9100" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "13786651648" 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713205293" 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "2395426816" 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "0" 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "instance": "10.142.0.107:9100" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "instance": "10.142.0.43:9100" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "5818.533333333334" 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "pod": "ds-cts-1" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "0" 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "instance": "10.142.0.107:9100" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "instance": "10.142.0.43:9100" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "0.00033333333333333365" 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "pod": "ds-idrepo-1" 19:21:33 DEBUG "pod": "ds-idrepo-1" 19:21:33 DEBUG }, 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "pod": "ds-idrepo-0" 19:21:33 DEBUG "pod": "ds-idrepo-0" 19:21:33 DEBUG }, 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "0" 19:21:33 DEBUG "0" 19:21:33 DEBUG ] 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "pod": "ds-cts-1" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "0" 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "pod": "ds-idrepo-1" 19:21:33 DEBUG "pod": "ds-cts-1" 19:21:33 DEBUG }, 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "pod": "ds-cts-2" 19:21:33 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 19:21:33 DEBUG }, 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "0" 19:21:33 DEBUG "0" 19:21:33 DEBUG ] 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "444266.5" 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "pod": "ds-idrepo-1" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "pod": "ds-idrepo-0" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "0" 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205293" 19:21:33 INFO [http_cmd]: http status code OK 19:21:33 DEBUG --- status code --- 19:21:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:21:33 DEBUG --- http response --- 19:21:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:21:33 DEBUG ----- output ----- 19:21:33 DEBUG { 19:21:33 DEBUG "status": "success", 19:21:33 DEBUG "data": { 19:21:33 DEBUG "resultType": "vector", 19:21:33 DEBUG "result": [ 19:21:33 DEBUG { 19:21:33 DEBUG "metric": { 19:21:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG [...] 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG 19:21:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:21:33 DEBUG }, 19:21:33 DEBUG "value": [ 19:21:33 DEBUG 1713205293, 19:21:33 DEBUG "2499.133333333333" 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG ] 19:21:33 DEBUG } 19:21:33 DEBUG } 19:21:34 INFO 19:21:34 INFO [loop_until]: kubectl --namespace=xlou top pods 19:21:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:21:35 INFO [loop_until]: OK (rc = 0) 19:21:35 DEBUG --- stdout --- 19:21:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 14139m 9014Mi am-7d597cf7d6-8vr8f 13045m 8881Mi am-7d597cf7d6-9wz9h 17007m 9308Mi ds-cts-0 16798m 8395Mi ds-cts-1 12457m 8470Mi ds-cts-2 11566m 8594Mi ds-idrepo-0 13853m 4941Mi ds-idrepo-1 2240m 4341Mi ds-idrepo-2 2083m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 4m 1222Mi lodemon-7c47b75958-dgfm9 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7592m 2212Mi 19:21:35 DEBUG --- stderr --- 19:21:35 DEBUG 19:21:35 INFO 19:21:35 INFO [loop_until]: kubectl --namespace=xlou top node 19:21:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:21:35 INFO [loop_until]: OK (rc = 0) 19:21:35 DEBUG --- stdout --- 19:21:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 59m 0% 5088Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 11926m 39% 10707Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 14489m 48% 11100Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 21404m 71% 3429Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 2249m 7% 7123Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 17630m 59% 12695Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2977Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 13930m 46% 10392Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 2267m 7% 6268Mi 5% gke-xlou-cdm-ds-0f601751-j488 11807m 39% 10137Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2442Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 18846m 63% 11049Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 14151m 47% 7088Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 7810m 49% 6014Mi 10% 19:21:35 DEBUG --- stderr --- 19:21:35 DEBUG 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713205353" 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "1155.6" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 ERROR [http_cmd]: ERROR 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 ERROR --- status code --- 19:22:33 DEBUG --- status code --- 19:22:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 ERROR --- http response --- 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:22:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "2488.133333333333" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713205353" 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "8.035514991591926" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205353" 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "pod": "ds-idrepo-1" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "pod": "ds-idrepo-0" 19:22:33 DEBUG }, 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "0" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205353" 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713205353" 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:33 DEBUG --- http response --- 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG { 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG { 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG { 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:22:33 DEBUG "data": { 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG { 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG "data": { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "data": { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:22:33 DEBUG "metric": { 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205353" 19:22:33 DEBUG "instance": "10.142.0.107:9100" 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "instance": "10.142.0.107:9100" 19:22:33 DEBUG }, 19:22:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:22:33 DEBUG }, 19:22:33 DEBUG "result": [ 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:33 DEBUG }, 19:22:33 DEBUG "result": [ 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG { 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG "value": [ 19:22:33 DEBUG { 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG "metric": { 19:22:33 DEBUG { 19:22:33 DEBUG 19:22:33 DEBUG "metric": { 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "pod": "ds-idrepo-1" 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG 19:22:33 DEBUG "pod": "ds-idrepo-1" 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG }, 19:22:33 DEBUG "data": { 19:22:33 DEBUG 19:22:33 DEBUG }, 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG "value": [ 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG [...] 19:22:33 DEBUG "value": [ 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "result": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG { 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "metric": { 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:22:33 DEBUG 19:22:33 DEBUG "instance": "10.142.0.43:9100" 19:22:33 DEBUG [...] 19:22:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:22:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:22:33 DEBUG [...] 19:22:33 DEBUG "instance": "10.142.0.43:9100" 19:22:33 DEBUG }, 19:22:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:22:33 DEBUG }, 19:22:33 DEBUG 19:22:33 DEBUG }, 19:22:33 DEBUG }, 19:22:33 DEBUG 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG "value": [ 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "0.078" 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "0.00026666666666651885" 19:22:33 DEBUG "pod": "ds-idrepo-0" 19:22:33 DEBUG 19:22:33 DEBUG "1154.9333333333334" 19:22:33 DEBUG "pod": "ds-cts-2" 19:22:33 DEBUG "6584.066666666666" 19:22:33 DEBUG ] 19:22:33 DEBUG "0" 19:22:33 DEBUG ] 19:22:33 DEBUG }, 19:22:33 DEBUG 19:22:33 DEBUG ] 19:22:33 DEBUG }, 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG "value": [ 19:22:33 DEBUG [...] 19:22:33 DEBUG } 19:22:33 DEBUG "value": [ 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG 19:22:33 DEBUG ] 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG "0" 19:22:33 DEBUG 19:22:33 DEBUG } 19:22:33 DEBUG "0.00007912000000000001" 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:22:33 DEBUG } 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG ] 19:22:33 DEBUG }, 19:22:33 DEBUG ] 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG } 19:22:33 DEBUG "value": [ 19:22:33 DEBUG } 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG } 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG } 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG "3908296704" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "instance": "10.142.0.107:9100" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "instance": "10.142.0.43:9100" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "13787333632" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "0" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205353" 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205353" 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "instance": "10.142.0.107:9100" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "instance": "10.142.0.43:9100" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "6215.6" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "9.79614216901015" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG { 19:22:33 DEBUG "result": [ 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG { 19:22:33 DEBUG "data": { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "pod": "ds-idrepo-1" 19:22:33 DEBUG "result": [ 19:22:33 DEBUG }, 19:22:33 DEBUG { 19:22:33 DEBUG "value": [ 19:22:33 DEBUG "metric": { 19:22:33 DEBUG 19:22:33 DEBUG "instance": "10.142.0.107:9100" 19:22:33 DEBUG 19:22:33 DEBUG }, 19:22:33 DEBUG 19:22:33 DEBUG "value": [ 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG [...] 19:22:33 DEBUG "pod": "ds-idrepo-0" 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG 19:22:33 DEBUG }, 19:22:33 DEBUG 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "instance": "10.142.0.43:9100" 19:22:33 DEBUG { 19:22:33 DEBUG "0" 19:22:33 DEBUG }, 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG ] 19:22:33 DEBUG "value": [ 19:22:33 DEBUG "data": { 19:22:33 DEBUG } 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG ] 19:22:33 DEBUG "0" 19:22:33 DEBUG "result": [ 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG { 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 DEBUG "metric": { 19:22:33 DEBUG ] 19:22:33 DEBUG "pod": "ds-idrepo-1" 19:22:33 DEBUG } 19:22:33 DEBUG }, 19:22:33 DEBUG } 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "pod": "ds-idrepo-0" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "0" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "255003399.80509537" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "pod": "ds-idrepo-1" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG [...] 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "pod": "ds-cts-2" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "0" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "9479705.30419045" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "401148" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "pod": "ds-idrepo-1" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "pod": "ds-idrepo-0" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "0" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205353" 19:22:33 INFO [http_cmd]: http status code OK 19:22:33 DEBUG --- status code --- 19:22:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:22:33 DEBUG --- http response --- 19:22:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:22:33 DEBUG ----- output ----- 19:22:33 DEBUG { 19:22:33 DEBUG "status": "success", 19:22:33 DEBUG "data": { 19:22:33 DEBUG "resultType": "vector", 19:22:33 DEBUG "result": [ 19:22:33 DEBUG { 19:22:33 DEBUG "metric": { 19:22:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG [...] 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG 19:22:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:22:33 DEBUG }, 19:22:33 DEBUG "value": [ 19:22:33 DEBUG 1713205353, 19:22:33 DEBUG "1155" 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG ] 19:22:33 DEBUG } 19:22:33 DEBUG } 19:22:35 INFO 19:22:35 INFO [loop_until]: kubectl --namespace=xlou top pods 19:22:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:22:35 INFO [loop_until]: OK (rc = 0) 19:22:35 DEBUG --- stdout --- 19:22:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 7636m 8844Mi am-7d597cf7d6-8vr8f 7273m 8879Mi am-7d597cf7d6-9wz9h 9566m 8731Mi ds-cts-0 11685m 8372Mi ds-cts-1 8363m 8434Mi ds-cts-2 8207m 8578Mi ds-idrepo-0 7317m 4941Mi ds-idrepo-1 1089m 4340Mi ds-idrepo-2 1098m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8102m 4548Mi 19:22:35 DEBUG --- stderr --- 19:22:35 DEBUG 19:22:35 INFO 19:22:35 INFO [loop_until]: kubectl --namespace=xlou top node 19:22:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:22:35 INFO [loop_until]: OK (rc = 0) 19:22:35 DEBUG --- stdout --- 19:22:35 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5091Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8592m 28% 10710Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7963m 26% 10931Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 9883m 33% 3606Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1182m 3% 7129Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 12736m 42% 12701Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 60m 0% 2968Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 7745m 25% 10409Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1200m 4% 6261Mi 5% gke-xlou-cdm-ds-0f601751-j488 8437m 28% 10125Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2447Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 10263m 34% 10403Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 7312m 24% 7095Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8196m 51% 8082Mi 13% 19:22:35 DEBUG --- stderr --- 19:22:35 DEBUG 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713205413" 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:23:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:23:33 DEBUG }, 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:23:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:23:33 DEBUG }, 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "2438.633333333333" 19:23:33 DEBUG "1202.5666666666666" 19:23:33 DEBUG ] 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713205413" 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205413" 19:23:33 ERROR [http_cmd]: ERROR 19:23:33 ERROR --- status code --- 19:23:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:23:33 ERROR --- http response --- 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:23:33 DEBUG --- status code --- 19:23:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "pod": "ds-idrepo-1" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "pod": "ds-idrepo-0" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "0" 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "8.027952748989563" 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "instance": "10.142.0.107:9100" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "instance": "10.142.0.43:9100" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "0.0004666666666665264" 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "0" 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205413" 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205413" 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 DEBUG --- status code --- 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205413" 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713205413" 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "pod": "ds-idrepo-1" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "pod": "ds-idrepo-0" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "0" 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "0.078" 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG { 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "1202.3" 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "instance": "10.142.0.107:9100" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "instance": "10.142.0.43:9100" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "0.00033333333333333365" 19:23:33 DEBUG ] 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG ] 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG } 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG } 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG { 19:23:33 DEBUG { 19:23:33 DEBUG { 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG "data": { 19:23:33 DEBUG "data": { 19:23:33 DEBUG "data": { 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:33 DEBUG "resultType": "vector", 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG "result": [ 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG "result": [ 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG { 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG { 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "metric": { 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "data": { 19:23:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:23:33 DEBUG "pod": "ds-idrepo-1" 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:33 DEBUG "instance": "10.142.0.107:9100" 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG }, 19:23:33 DEBUG }, 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG }, 19:23:33 DEBUG "result": [ 19:23:33 DEBUG "value": [ 19:23:33 DEBUG "value": [ 19:23:33 DEBUG { 19:23:33 DEBUG "value": [ 19:23:33 DEBUG { 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG 19:23:33 DEBUG "metric": { 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "data": { 19:23:33 DEBUG 19:23:33 DEBUG "instance": "10.142.0.107:9100" 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG 19:23:33 DEBUG }, 19:23:33 DEBUG [...] 19:23:33 DEBUG [...] 19:23:33 DEBUG "result": [ 19:23:33 DEBUG [...] 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG { 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "metric": { 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:23:33 DEBUG "pod": "ds-cts-2" 19:23:33 DEBUG }, 19:23:33 DEBUG "instance": "10.142.0.43:9100" 19:23:33 DEBUG [...] 19:23:33 DEBUG }, 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG }, 19:23:33 DEBUG 19:23:33 DEBUG "value": [ 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG 19:23:33 DEBUG "0" 19:23:33 DEBUG "0.00015824000000000003" 19:23:33 DEBUG 19:23:33 DEBUG "6633.833333333333" 19:23:33 DEBUG "instance": "10.142.0.43:9100" 19:23:33 DEBUG ] 19:23:33 DEBUG ] 19:23:33 DEBUG [...] 19:23:33 DEBUG ] 19:23:33 DEBUG }, 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 DEBUG 19:23:33 DEBUG } 19:23:33 DEBUG "value": [ 19:23:33 DEBUG ] 19:23:33 DEBUG ] 19:23:33 DEBUG 19:23:33 DEBUG ] 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 DEBUG 19:23:33 DEBUG } 19:23:33 DEBUG "13788036096" 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG }, 19:23:33 DEBUG } 19:23:33 DEBUG "value": [ 19:23:33 DEBUG ] 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG } 19:23:33 DEBUG "6788349952" 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205413" 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "instance": "10.142.0.107:9100" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "instance": "10.142.0.43:9100" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "5901.433333333333" 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "0.0009677604581048096" 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "pod": "ds-idrepo-1" 19:23:33 DEBUG "pod": "ds-idrepo-1" 19:23:33 DEBUG }, 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "pod": "ds-idrepo-0" 19:23:33 DEBUG "pod": "ds-idrepo-0" 19:23:33 DEBUG }, 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "0" 19:23:33 DEBUG "0" 19:23:33 DEBUG ] 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "1984755.389919462" 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG { 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (116 lines): 19:23:33 DEBUG "result": [ 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "pod": "ds-idrepo-1" 19:23:33 DEBUG "data": { 19:23:33 DEBUG }, 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "value": [ 19:23:33 DEBUG "result": [ 19:23:33 DEBUG 19:23:33 DEBUG { 19:23:33 DEBUG 19:23:33 DEBUG "metric": { 19:23:33 DEBUG 19:23:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:23:33 DEBUG [...] 19:23:33 DEBUG }, 19:23:33 DEBUG 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "pod": "ds-cts-2" 19:23:33 DEBUG 19:23:33 DEBUG }, 19:23:33 DEBUG [...] 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG 19:23:33 DEBUG "0" 19:23:33 DEBUG 19:23:33 DEBUG ] 19:23:33 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:23:33 DEBUG } 19:23:33 DEBUG }, 19:23:33 DEBUG ] 19:23:33 DEBUG "value": [ 19:23:33 DEBUG } 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG } 19:23:33 DEBUG "0" 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "327273" 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "pod": "ds-idrepo-1" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "pod": "ds-idrepo-0" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "0" 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205413" 19:23:33 INFO [http_cmd]: http status code OK 19:23:33 DEBUG --- status code --- 19:23:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:23:33 DEBUG --- http response --- 19:23:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:23:33 DEBUG ----- output ----- 19:23:33 DEBUG { 19:23:33 DEBUG "status": "success", 19:23:33 DEBUG "data": { 19:23:33 DEBUG "resultType": "vector", 19:23:33 DEBUG "result": [ 19:23:33 DEBUG { 19:23:33 DEBUG "metric": { 19:23:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG [...] 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG 19:23:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:23:33 DEBUG }, 19:23:33 DEBUG "value": [ 19:23:33 DEBUG 1713205413, 19:23:33 DEBUG "1202.2333333333333" 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG ] 19:23:33 DEBUG } 19:23:33 DEBUG } 19:23:35 INFO 19:23:35 INFO [loop_until]: kubectl --namespace=xlou top pods 19:23:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:23:35 INFO [loop_until]: OK (rc = 0) 19:23:35 DEBUG --- stdout --- 19:23:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 7760m 8846Mi am-7d597cf7d6-8vr8f 7143m 8883Mi am-7d597cf7d6-9wz9h 9910m 8730Mi ds-cts-0 11958m 8382Mi ds-cts-1 7935m 8455Mi ds-cts-2 8438m 8579Mi ds-idrepo-0 7555m 4941Mi ds-idrepo-1 1120m 4341Mi ds-idrepo-2 1101m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 5m 1023Mi idm-b85f9d978-rl76f 4m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7940m 6762Mi 19:23:35 DEBUG --- stderr --- 19:23:35 DEBUG 19:23:35 INFO 19:23:35 INFO [loop_until]: kubectl --namespace=xlou top node 19:23:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:23:36 INFO [loop_until]: OK (rc = 0) 19:23:36 DEBUG --- stdout --- 19:23:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 5085Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 8446m 28% 10691Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 8115m 27% 10930Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10493m 35% 3606Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1241m 4% 7133Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 12615m 42% 12724Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 2968Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 7655m 25% 10405Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1194m 3% 6261Mi 5% gke-xlou-cdm-ds-0f601751-j488 8034m 26% 10132Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 65m 0% 2449Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 10539m 35% 10391Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 7401m 24% 7099Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 7951m 50% 10014Mi 17% 19:23:36 DEBUG --- stderr --- 19:23:36 DEBUG 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713205473" 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 ERROR [http_cmd]: ERROR 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG --- status code --- 19:24:33 ERROR --- status code --- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG --- http response --- 19:24:33 ERROR --- http response --- 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205473" 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:24:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG ----- output ----- 19:24:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:24:33 DEBUG { 19:24:33 DEBUG { 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "data": { 19:24:33 DEBUG "data": { 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "result": [ 19:24:33 DEBUG "result": [ 19:24:33 DEBUG { 19:24:33 DEBUG { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:24:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:24:33 DEBUG }, 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG [...] 19:24:33 DEBUG [...] 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:24:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:24:33 DEBUG }, 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "2419.3333333333335" 19:24:33 DEBUG "1122.7" 19:24:33 DEBUG ] 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG --- status code --- 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG { 19:24:33 DEBUG { 19:24:33 DEBUG { 19:24:33 DEBUG { 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "data": { 19:24:33 DEBUG "data": { 19:24:33 DEBUG "data": { 19:24:33 DEBUG "data": { 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "result": [ 19:24:33 DEBUG "result": [ 19:24:33 DEBUG "result": [ 19:24:33 DEBUG "result": [ 19:24:33 DEBUG { 19:24:33 DEBUG { 19:24:33 DEBUG { 19:24:33 DEBUG { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "pod": "ds-idrepo-1" 19:24:33 DEBUG "pod": "ds-idrepo-1" 19:24:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:24:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:24:33 DEBUG }, 19:24:33 DEBUG }, 19:24:33 DEBUG }, 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG "value": [ 19:24:33 DEBUG "value": [ 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG [...] 19:24:33 DEBUG [...] 19:24:33 DEBUG [...] 19:24:33 DEBUG [...] 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG "pod": "ds-idrepo-0" 19:24:33 DEBUG "pod": "ds-idrepo-0" 19:24:33 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 19:24:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:24:33 DEBUG }, 19:24:33 DEBUG }, 19:24:33 DEBUG }, 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG "value": [ 19:24:33 DEBUG "value": [ 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "0" 19:24:33 DEBUG "0" 19:24:33 DEBUG "0" 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG "8.033191826239127" 19:24:33 DEBUG ] 19:24:33 DEBUG ] 19:24:33 DEBUG ] 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG ] 19:24:33 DEBUG ] 19:24:33 DEBUG { 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 DEBUG "data": { 19:24:33 DEBUG } 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "result": [ 19:24:33 DEBUG { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "instance": "10.142.0.107:9100" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG [...] 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG "instance": "10.142.0.43:9100" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "0.00026666666666651885" 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205473" 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 DEBUG --- status code --- 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG { 19:24:33 DEBUG { 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "data": { 19:24:33 DEBUG "data": { 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "result": [ 19:24:33 DEBUG "result": [ 19:24:33 DEBUG { 19:24:33 DEBUG { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:24:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:24:33 DEBUG }, 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG 19:24:33 DEBUG [...] 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:33 DEBUG [...] 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG 19:24:33 DEBUG { 19:24:33 DEBUG 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:24:33 DEBUG "data": { 19:24:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:24:33 DEBUG }, 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG "result": [ 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG { 19:24:33 DEBUG { 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "1122.9333333333334" 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "0.078" 19:24:33 DEBUG ] 19:24:33 DEBUG "data": { 19:24:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:24:33 DEBUG ] 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG } 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG }, 19:24:33 DEBUG } 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG ] 19:24:33 DEBUG "result": [ 19:24:33 DEBUG "value": [ 19:24:33 DEBUG ] 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG } 19:24:33 DEBUG { 19:24:33 DEBUG 19:24:33 DEBUG } 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG } 19:24:33 DEBUG "metric": { 19:24:33 DEBUG 19:24:33 DEBUG } 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:33 DEBUG "instance": "10.142.0.107:9100" 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG }, 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG [...] 19:24:33 DEBUG { 19:24:33 DEBUG "value": [ 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG 19:24:33 DEBUG "data": { 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG 19:24:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:24:33 DEBUG ----- output ----- 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG "result": [ 19:24:33 DEBUG [...] 19:24:33 DEBUG }, 19:24:33 DEBUG { 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG { 19:24:33 DEBUG 19:24:33 DEBUG "value": [ 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG "metric": { 19:24:33 DEBUG 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "data": { 19:24:33 DEBUG --- http response --- 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG "instance": "10.142.0.107:9100" 19:24:33 DEBUG 19:24:33 DEBUG "0" 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG }, 19:24:33 DEBUG "instance": "10.142.0.43:9100" 19:24:33 DEBUG ] 19:24:33 DEBUG "result": [ 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG "value": [ 19:24:33 DEBUG }, 19:24:33 DEBUG } 19:24:33 DEBUG { 19:24:33 DEBUG { 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG 19:24:33 DEBUG "value": [ 19:24:33 DEBUG ] 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG } 19:24:33 DEBUG "instance": "10.142.0.107:9100" 19:24:33 DEBUG "data": { 19:24:33 DEBUG 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG "0" 19:24:33 DEBUG } 19:24:33 DEBUG }, 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG [...] 19:24:33 DEBUG { 19:24:33 DEBUG ] 19:24:33 DEBUG "value": [ 19:24:33 DEBUG "result": [ 19:24:33 DEBUG 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG } 19:24:33 DEBUG 19:24:33 DEBUG { 19:24:33 DEBUG 19:24:33 DEBUG "data": { 19:24:33 DEBUG ] 19:24:33 DEBUG 19:24:33 DEBUG "metric": { 19:24:33 DEBUG 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG } 19:24:33 DEBUG 19:24:33 DEBUG "pod": "ds-idrepo-1" 19:24:33 DEBUG "instance": "10.142.0.43:9100" 19:24:33 DEBUG "result": [ 19:24:33 DEBUG } 19:24:33 DEBUG [...] 19:24:33 DEBUG }, 19:24:33 DEBUG }, 19:24:33 DEBUG { 19:24:33 DEBUG 19:24:33 DEBUG "value": [ 19:24:33 DEBUG "value": [ 19:24:33 DEBUG "metric": { 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG "5157.2" 19:24:33 DEBUG }, 19:24:33 DEBUG "instance": "10.142.0.43:9100" 19:24:33 DEBUG 19:24:33 DEBUG ] 19:24:33 DEBUG "value": [ 19:24:33 DEBUG }, 19:24:33 DEBUG [...] 19:24:33 DEBUG } 19:24:33 DEBUG 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG ] 19:24:33 DEBUG 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG 19:24:33 DEBUG } 19:24:33 DEBUG 19:24:33 DEBUG "13788740608" 19:24:33 DEBUG 19:24:33 DEBUG } 19:24:33 DEBUG [...] 19:24:33 DEBUG ] 19:24:33 DEBUG "pod": "ds-cts-2" 19:24:33 DEBUG 19:24:33 DEBUG } 19:24:33 DEBUG }, 19:24:33 DEBUG 19:24:33 DEBUG ] 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG } 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:24:33 DEBUG } 19:24:33 DEBUG "0" 19:24:33 DEBUG }, 19:24:33 DEBUG ] 19:24:33 DEBUG "value": [ 19:24:33 DEBUG } 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG ] 19:24:33 DEBUG "8074801152" 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG { 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "data": { 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "result": [ 19:24:33 DEBUG { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "instance": "10.142.0.107:9100" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG [...] 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG "instance": "10.142.0.43:9100" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "6422.700000000001" 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG { 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "data": { 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "result": [ 19:24:33 DEBUG { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG [...] 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG "pod": "end-user-ui-58878d7668-zg4h2" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "0" 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG { 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "data": { 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "result": [ 19:24:33 DEBUG { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "pod": "ds-idrepo-1" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG [...] 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG "pod": "ds-cts-2" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "0" 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG { 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "data": { 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "result": [ 19:24:33 DEBUG { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "pod": "ds-idrepo-1" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG [...] 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG "pod": "ds-idrepo-0" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "0" 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG { 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "data": { 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "result": [ 19:24:33 DEBUG { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "pod": "ds-idrepo-1" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG [...] 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG "pod": "ds-idrepo-0" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "0" 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG { 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "data": { 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "result": [ 19:24:33 DEBUG { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG [...] 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG "pod": "end-user-ui-58878d7668-zg4h2" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "0" 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG { 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "data": { 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "result": [ 19:24:33 DEBUG { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG [...] 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "0" 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG { 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "data": { 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "result": [ 19:24:33 DEBUG { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG [...] 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "252225.5" 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG { 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "data": { 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "result": [ 19:24:33 DEBUG { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "pod": "ds-idrepo-1" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG [...] 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG "pod": "ds-idrepo-0" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "0" 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205473" 19:24:33 INFO [http_cmd]: http status code OK 19:24:33 DEBUG --- status code --- 19:24:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:24:33 DEBUG --- http response --- 19:24:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:24:33 DEBUG ----- output ----- 19:24:33 DEBUG { 19:24:33 DEBUG "status": "success", 19:24:33 DEBUG "data": { 19:24:33 DEBUG "resultType": "vector", 19:24:33 DEBUG "result": [ 19:24:33 DEBUG { 19:24:33 DEBUG "metric": { 19:24:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG [...] 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG 19:24:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:24:33 DEBUG }, 19:24:33 DEBUG "value": [ 19:24:33 DEBUG 1713205473, 19:24:33 DEBUG "1123.0333333333333" 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG ] 19:24:33 DEBUG } 19:24:33 DEBUG } 19:24:35 INFO 19:24:35 INFO [loop_until]: kubectl --namespace=xlou top pods 19:24:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:24:35 INFO [loop_until]: OK (rc = 0) 19:24:35 DEBUG --- stdout --- 19:24:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 7502m 8849Mi am-7d597cf7d6-8vr8f 6912m 8882Mi am-7d597cf7d6-9wz9h 8753m 8729Mi ds-cts-0 11246m 8402Mi ds-cts-1 8039m 8446Mi ds-cts-2 7823m 8615Mi ds-idrepo-0 7155m 4940Mi ds-idrepo-1 1086m 4341Mi ds-idrepo-2 1020m 4112Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1023Mi idm-b85f9d978-rl76f 4m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8162m 7799Mi 19:24:35 DEBUG --- stderr --- 19:24:35 DEBUG 19:24:36 INFO 19:24:36 INFO [loop_until]: kubectl --namespace=xlou top node 19:24:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:24:36 INFO [loop_until]: OK (rc = 0) 19:24:36 DEBUG --- stdout --- 19:24:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5083Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7863m 26% 10704Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7804m 26% 10927Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 9860m 33% 3623Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1216m 4% 7134Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 11941m 39% 12694Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2973Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 7565m 25% 10413Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 1149m 3% 6267Mi 5% gke-xlou-cdm-ds-0f601751-j488 7925m 26% 10138Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2449Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 10144m 33% 10385Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 7292m 24% 7097Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8293m 52% 10889Mi 18% 19:24:36 DEBUG --- stderr --- 19:24:36 DEBUG 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 ERROR [http_cmd]: ERROR 19:25:33 ERROR --- status code --- 19:25:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:25:33 ERROR --- http response --- 19:25:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:25:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205533" 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "pod": "ds-idrepo-1" 19:25:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:25:33 DEBUG }, 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "pod": "ds-idrepo-0" 19:25:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:25:33 DEBUG }, 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "0" 19:25:33 DEBUG "1157.7666666666667" 19:25:33 DEBUG ] 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG { 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "1100.6" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "0" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "pod": "ds-idrepo-1" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "pod": "ds-idrepo-0" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "0" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713205533" 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "instance": "10.142.0.107:9100" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "instance": "10.142.0.43:9100" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "0.00020000000000000757" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "8.03554259320743" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205533" 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "0.078" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "instance": "10.142.0.107:9100" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "instance": "10.142.0.43:9100" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "13789275136" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "1100.5333333333333" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "pod": "ds-idrepo-1" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "pod": "ds-cts-2" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "0.00007912000000000001" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713205533" 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "0" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "instance": "10.142.0.107:9100" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "instance": "10.142.0.43:9100" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "5452.966666666666" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "pod": "ds-idrepo-1" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "pod": "ds-cts-2" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "0" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:25:33 DEBUG "instance": "10.142.0.107:9100" 19:25:33 DEBUG }, 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:25:33 DEBUG "instance": "10.142.0.43:9100" 19:25:33 DEBUG }, 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "8374390784" 19:25:33 DEBUG "0" 19:25:33 DEBUG ] 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "instance": "10.142.0.107:9100" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "instance": "10.142.0.43:9100" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "6705.733333333334" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "0" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG ----- output ----- 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG { 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG "data": { 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG { 19:25:33 DEBUG "result": [ 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG { 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG "data": { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "pod": "ds-idrepo-1" 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG }, 19:25:33 DEBUG "data": { 19:25:33 DEBUG { 19:25:33 DEBUG "value": [ 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "metric": { 19:25:33 DEBUG 19:25:33 DEBUG "result": [ 19:25:33 DEBUG "pod": "ds-idrepo-1" 19:25:33 DEBUG 19:25:33 DEBUG { 19:25:33 DEBUG }, 19:25:33 DEBUG 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "value": [ 19:25:33 DEBUG [...] 19:25:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG }, 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG "pod": "ds-idrepo-0" 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG }, 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "value": [ 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG 19:25:33 DEBUG "pod": "ds-idrepo-0" 19:25:33 DEBUG "0" 19:25:33 DEBUG 19:25:33 DEBUG }, 19:25:33 DEBUG ] 19:25:33 DEBUG 19:25:33 DEBUG "value": [ 19:25:33 DEBUG } 19:25:33 DEBUG "pod": "idm-b85f9d978-j5pbm" 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG ] 19:25:33 DEBUG }, 19:25:33 DEBUG "0" 19:25:33 DEBUG } 19:25:33 DEBUG "value": [ 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG } 19:25:33 DEBUG "531.7753975981824" 19:25:33 DEBUG ] 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "0" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "208802.5" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205533" 19:25:33 INFO [http_cmd]: http status code OK 19:25:33 DEBUG --- status code --- 19:25:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:33 DEBUG --- http response --- 19:25:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:25:33 DEBUG ----- output ----- 19:25:33 DEBUG { 19:25:33 DEBUG "status": "success", 19:25:33 DEBUG "data": { 19:25:33 DEBUG "resultType": "vector", 19:25:33 DEBUG "result": [ 19:25:33 DEBUG { 19:25:33 DEBUG "metric": { 19:25:33 DEBUG "pod": "ds-idrepo-1" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG [...] 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG 19:25:33 DEBUG "pod": "ds-idrepo-0" 19:25:33 DEBUG }, 19:25:33 DEBUG "value": [ 19:25:33 DEBUG 1713205533, 19:25:33 DEBUG "0" 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG ] 19:25:33 DEBUG } 19:25:33 DEBUG } 19:25:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205534" 19:25:34 INFO [http_cmd]: http status code OK 19:25:34 DEBUG --- status code --- 19:25:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:25:34 DEBUG --- http response --- 19:25:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:25:34 DEBUG ----- output ----- 19:25:34 DEBUG { 19:25:34 DEBUG "status": "success", 19:25:34 DEBUG "data": { 19:25:34 DEBUG "resultType": "vector", 19:25:34 DEBUG "result": [ 19:25:34 DEBUG { 19:25:34 DEBUG "metric": { 19:25:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:25:34 DEBUG }, 19:25:34 DEBUG "value": [ 19:25:34 DEBUG 19:25:34 DEBUG 19:25:34 DEBUG 19:25:34 DEBUG [...] 19:25:34 DEBUG 19:25:34 DEBUG 19:25:34 DEBUG 19:25:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:25:34 DEBUG }, 19:25:34 DEBUG "value": [ 19:25:34 DEBUG 1713205534, 19:25:34 DEBUG "1100.5333333333333" 19:25:34 DEBUG ] 19:25:34 DEBUG } 19:25:34 DEBUG ] 19:25:34 DEBUG } 19:25:34 DEBUG } 19:25:35 INFO 19:25:35 INFO [loop_until]: kubectl --namespace=xlou top pods 19:25:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:25:35 INFO [loop_until]: OK (rc = 0) 19:25:35 DEBUG --- stdout --- 19:25:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5751m 8845Mi am-7d597cf7d6-8vr8f 5473m 8810Mi am-7d597cf7d6-9wz9h 7256m 8725Mi ds-cts-0 7080m 8400Mi ds-cts-1 4835m 8439Mi ds-cts-2 4649m 8606Mi ds-idrepo-0 6186m 4941Mi ds-idrepo-1 968m 4340Mi ds-idrepo-2 903m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1023Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7858m 8036Mi 19:25:35 DEBUG --- stderr --- 19:25:35 DEBUG 19:25:36 INFO 19:25:36 INFO [loop_until]: kubectl --namespace=xlou top node 19:25:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:25:36 INFO [loop_until]: OK (rc = 0) 19:25:36 DEBUG --- stdout --- 19:25:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 5086Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4951m 16% 10696Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6326m 21% 10938Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 9515m 31% 3616Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1053m 3% 7140Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 7454m 24% 12707Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 66m 0% 2975Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 5915m 19% 10348Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 992m 3% 6261Mi 5% gke-xlou-cdm-ds-0f601751-j488 4579m 15% 10154Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 63m 0% 2436Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 8107m 27% 10387Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 6028m 20% 7096Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 8055m 50% 11063Mi 18% 19:25:36 DEBUG --- stderr --- 19:25:36 DEBUG 19:25:51 WARNING File /tmp/lodemon_data-20240415_184321/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [15/Apr/2024 19:25:51] "GET /monitoring/average?start_time=20240415_181744&stop_time=20240415_182350 HTTP/1.1" 200 - 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "instance": "10.142.0.107:9100" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "instance": "10.142.0.43:9100" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "0.00026666666666651885" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "1203.5333333333333" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "2.6191333558248013" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 ERROR [http_cmd]: ERROR 19:26:33 ERROR --- status code --- 19:26:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:26:33 ERROR --- http response --- 19:26:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:26:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "0" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "pod": "ds-idrepo-1" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "pod": "ds-idrepo-0" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "0" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "pod": "ds-idrepo-1" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "pod": "ds-idrepo-0" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "0" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "0" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205593" 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "0.078" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "instance": "10.142.0.107:9100" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "instance": "10.142.0.43:9100" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "13789940736" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:26:33 DEBUG "pod": "ds-idrepo-1" 19:26:33 DEBUG }, 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:26:33 DEBUG "pod": "ds-cts-2" 19:26:33 DEBUG }, 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "0" 19:26:33 DEBUG "0.00005311333333333334" 19:26:33 DEBUG ] 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713205593" 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "0.006818015197061444" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "0" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "113065984" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205593" 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:33 DEBUG "data": { 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG { 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG "result": [ 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG { 19:26:33 DEBUG "data": { 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG "instance": "10.142.0.107:9100" 19:26:33 DEBUG "result": [ 19:26:33 DEBUG }, 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:26:33 DEBUG { 19:26:33 DEBUG "value": [ 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG "metric": { 19:26:33 DEBUG 19:26:33 DEBUG { 19:26:33 DEBUG "instance": "10.142.0.107:9100" 19:26:33 DEBUG 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG }, 19:26:33 DEBUG 19:26:33 DEBUG "data": { 19:26:33 DEBUG "value": [ 19:26:33 DEBUG [...] 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "result": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG { 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "metric": { 19:26:33 DEBUG [...] 19:26:33 DEBUG "instance": "10.142.0.43:9100" 19:26:33 DEBUG "pod": "ds-idrepo-1" 19:26:33 DEBUG 19:26:33 DEBUG }, 19:26:33 DEBUG }, 19:26:33 DEBUG 19:26:33 DEBUG "value": [ 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG 19:26:33 DEBUG "instance": "10.142.0.43:9100" 19:26:33 DEBUG "5397.533333333333" 19:26:33 DEBUG 19:26:33 DEBUG }, 19:26:33 DEBUG ] 19:26:33 DEBUG 19:26:33 DEBUG "value": [ 19:26:33 DEBUG } 19:26:33 DEBUG [...] 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG ] 19:26:33 DEBUG 19:26:33 DEBUG "6615.866666666667" 19:26:33 DEBUG } 19:26:33 DEBUG 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG 19:26:33 DEBUG } 19:26:33 DEBUG "pod": "ds-cts-2" 19:26:33 DEBUG ] 19:26:33 DEBUG }, 19:26:33 DEBUG } 19:26:33 DEBUG "value": [ 19:26:33 DEBUG } 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "0" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "instance": "10.142.0.107:9100" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "instance": "10.142.0.43:9100" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "0" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "pod": "ds-idrepo-1" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "pod": "ds-idrepo-0" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "0" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "pod": "ds-idrepo-1" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "pod": "ds-idrepo-0" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "0" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "0" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:33 INFO [http_cmd]: http status code OK 19:26:33 DEBUG --- status code --- 19:26:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:33 DEBUG --- http response --- 19:26:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:26:33 DEBUG ----- output ----- 19:26:33 DEBUG { 19:26:33 DEBUG "status": "success", 19:26:33 DEBUG "data": { 19:26:33 DEBUG "resultType": "vector", 19:26:33 DEBUG "result": [ 19:26:33 DEBUG { 19:26:33 DEBUG "metric": { 19:26:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG [...] 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG 19:26:33 DEBUG "pod": "lodemon-7c47b75958-dgfm9" 19:26:33 DEBUG }, 19:26:33 DEBUG "value": [ 19:26:33 DEBUG 1713205593, 19:26:33 DEBUG "5025.922606359934" 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG ] 19:26:33 DEBUG } 19:26:33 DEBUG } 19:26:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205593" 19:26:34 INFO [http_cmd]: http status code OK 19:26:34 DEBUG --- status code --- 19:26:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:34 DEBUG --- http response --- 19:26:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:34 DEBUG ----- output ----- 19:26:34 DEBUG { 19:26:34 DEBUG "status": "success", 19:26:34 DEBUG "data": { 19:26:34 DEBUG "resultType": "vector", 19:26:34 DEBUG "result": [ 19:26:34 DEBUG { 19:26:34 DEBUG "metric": { 19:26:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:26:34 DEBUG }, 19:26:34 DEBUG "value": [ 19:26:34 DEBUG 19:26:34 DEBUG 19:26:34 DEBUG 19:26:34 DEBUG [...] 19:26:34 DEBUG 19:26:34 DEBUG 19:26:34 DEBUG 19:26:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:26:34 DEBUG }, 19:26:34 DEBUG "value": [ 19:26:34 DEBUG 1713205593, 19:26:34 DEBUG "160382" 19:26:34 DEBUG ] 19:26:34 DEBUG } 19:26:34 DEBUG ] 19:26:34 DEBUG } 19:26:34 DEBUG } 19:26:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205594" 19:26:34 INFO [http_cmd]: http status code OK 19:26:34 DEBUG --- status code --- 19:26:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:34 DEBUG --- http response --- 19:26:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:26:34 DEBUG ----- output ----- 19:26:34 DEBUG { 19:26:34 DEBUG "status": "success", 19:26:34 DEBUG "data": { 19:26:34 DEBUG "resultType": "vector", 19:26:34 DEBUG "result": [ 19:26:34 DEBUG { 19:26:34 DEBUG "metric": { 19:26:34 DEBUG "pod": "ds-idrepo-1" 19:26:34 DEBUG }, 19:26:34 DEBUG "value": [ 19:26:34 DEBUG 19:26:34 DEBUG 19:26:34 DEBUG 19:26:34 DEBUG [...] 19:26:34 DEBUG 19:26:34 DEBUG 19:26:34 DEBUG 19:26:34 DEBUG "pod": "ds-idrepo-0" 19:26:34 DEBUG }, 19:26:34 DEBUG "value": [ 19:26:34 DEBUG 1713205594, 19:26:34 DEBUG "0" 19:26:34 DEBUG ] 19:26:34 DEBUG } 19:26:34 DEBUG ] 19:26:34 DEBUG } 19:26:34 DEBUG } 19:26:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205594" 19:26:34 INFO [http_cmd]: http status code OK 19:26:34 DEBUG --- status code --- 19:26:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:26:34 DEBUG --- http response --- 19:26:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:26:34 DEBUG ----- output ----- 19:26:34 DEBUG { 19:26:34 DEBUG "status": "success", 19:26:34 DEBUG "data": { 19:26:34 DEBUG "resultType": "vector", 19:26:34 DEBUG "result": [ 19:26:34 DEBUG { 19:26:34 DEBUG "metric": { 19:26:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:26:34 DEBUG }, 19:26:34 DEBUG "value": [ 19:26:34 DEBUG 19:26:34 DEBUG 19:26:34 DEBUG 19:26:34 DEBUG [...] 19:26:34 DEBUG 19:26:34 DEBUG 19:26:34 DEBUG 19:26:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:26:34 DEBUG }, 19:26:34 DEBUG "value": [ 19:26:34 DEBUG 1713205594, 19:26:34 DEBUG "0" 19:26:34 DEBUG ] 19:26:34 DEBUG } 19:26:34 DEBUG ] 19:26:34 DEBUG } 19:26:34 DEBUG } 19:26:35 INFO 19:26:35 INFO [loop_until]: kubectl --namespace=xlou top pods 19:26:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:26:35 INFO [loop_until]: OK (rc = 0) 19:26:35 DEBUG --- stdout --- 19:26:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 1173m 8842Mi am-7d597cf7d6-8vr8f 1097m 8812Mi am-7d597cf7d6-9wz9h 1415m 8719Mi ds-cts-0 3613m 8392Mi ds-cts-1 2588m 8443Mi ds-cts-2 2529m 8596Mi ds-idrepo-0 668m 4941Mi ds-idrepo-1 105m 4340Mi ds-idrepo-2 96m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1022Mi idm-b85f9d978-rl76f 4m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 16Mi 19:26:35 DEBUG --- stderr --- 19:26:35 DEBUG 19:26:36 INFO 19:26:36 INFO [loop_until]: kubectl --namespace=xlou top node 19:26:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:26:36 INFO [loop_until]: OK (rc = 0) 19:26:36 DEBUG --- stdout --- 19:26:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 5095Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2560m 8% 10706Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1184m 3% 10979Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 98m 0% 3111Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 159m 0% 7142Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 3893m 13% 12704Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 69m 0% 2979Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1178m 3% 10366Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 158m 0% 6259Mi 5% gke-xlou-cdm-ds-0f601751-j488 2616m 8% 10147Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 92m 0% 2434Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 1617m 5% 10424Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 751m 2% 7101Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 76m 0% 3911Mi 6% 19:26:36 DEBUG --- stderr --- 19:26:36 DEBUG 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713205653" 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205653" 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "data": { 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG [...] 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG "1.3932639651582026" 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "data": { 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "instance": "10.142.0.107:9100" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG [...] 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG "instance": "10.142.0.43:9100" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG "0.0005333333333335114" 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713205653" 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG --- http response --- 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG { 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:33 DEBUG "data": { 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG { 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "data": { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "data": { 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "result": [ 19:27:33 DEBUG }, 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG "value": [ 19:27:33 DEBUG { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:27:33 DEBUG 19:27:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:27:33 DEBUG }, 19:27:33 DEBUG 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG [...] 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG [...] 19:27:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:27:33 DEBUG [...] 19:27:33 DEBUG 19:27:33 DEBUG }, 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG 19:27:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:27:33 DEBUG "0" 19:27:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:27:33 DEBUG }, 19:27:33 DEBUG ] 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG } 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG ] 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG "1125.3666666666666" 19:27:33 DEBUG } 19:27:33 DEBUG "0" 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 DEBUG ] 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 ERROR [http_cmd]: ERROR 19:27:33 ERROR --- status code --- 19:27:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:27:33 ERROR --- http response --- 19:27:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:27:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "data": { 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "pod": "ds-idrepo-1" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG [...] 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG "pod": "ds-idrepo-0" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG "0" 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "data": { 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "pod": "ds-idrepo-1" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG [...] 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG "pod": "ds-idrepo-0" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG "0" 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713205653" 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205653" 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "data": { 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG [...] 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG "0.078" 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "data": { 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "instance": "10.142.0.107:9100" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG [...] 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG "instance": "10.142.0.43:9100" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG "13790690304" 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "data": { 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG [...] 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG "544718848" 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG { 19:27:33 DEBUG { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "data": { 19:27:33 DEBUG "data": { 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "result": [ 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "pod": "ds-idrepo-1" 19:27:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:27:33 DEBUG }, 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG [...] 19:27:33 DEBUG [...] 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG "pod": "ds-cts-2" 19:27:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:27:33 DEBUG }, 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG "0.00003956000000000001" 19:27:33 DEBUG "0" 19:27:33 DEBUG ] 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG } 19:27:33 DEBUG ] 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "data": { 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG [...] 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG "1.3496195418302124" 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "data": { 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG [...] 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG "0" 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "data": { 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "pod": "ds-idrepo-1" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG [...] 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG "pod": "ds-idrepo-0" 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG "0" 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205653" 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205653" 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG { 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG "data": { 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG "result": [ 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG { 19:27:33 DEBUG { 19:27:33 DEBUG { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "data": { 19:27:33 DEBUG "instance": "10.142.0.107:9100" 19:27:33 DEBUG "data": { 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG }, 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "result": [ 19:27:33 DEBUG "value": [ 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG 19:27:33 DEBUG { 19:27:33 DEBUG "metric": { 19:27:33 DEBUG 19:27:33 DEBUG "metric": { 19:27:33 DEBUG "instance": "10.142.0.107:9100" 19:27:33 DEBUG 19:27:33 DEBUG "pod": "ds-idrepo-1" 19:27:33 DEBUG }, 19:27:33 DEBUG [...] 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG "instance": "10.142.0.43:9100" 19:27:33 DEBUG 19:27:33 DEBUG [...] 19:27:33 DEBUG }, 19:27:33 DEBUG [...] 19:27:33 DEBUG 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG 19:27:33 DEBUG 19:27:33 DEBUG "6530.866666666667" 19:27:33 DEBUG 19:27:33 DEBUG "instance": "10.142.0.43:9100" 19:27:33 DEBUG ] 19:27:33 DEBUG "pod": "ds-cts-2" 19:27:33 DEBUG }, 19:27:33 DEBUG } 19:27:33 DEBUG }, 19:27:33 DEBUG "value": [ 19:27:33 DEBUG ] 19:27:33 DEBUG "value": [ 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG } 19:27:33 DEBUG 1713205653, 19:27:33 DEBUG "6658.9" 19:27:33 DEBUG } 19:27:33 DEBUG "0" 19:27:33 DEBUG ] 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 DEBUG ] 19:27:33 DEBUG ] 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 DEBUG } 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205653" 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205653" 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG { 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG "data": { 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG "result": [ 19:27:33 DEBUG { 19:27:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:33 DEBUG "metric": { 19:27:33 DEBUG ----- output ----- 19:27:33 DEBUG "instance": "10.142.0.107:9100" 19:27:33 DEBUG { 19:27:33 DEBUG }, 19:27:33 DEBUG "status": "success", 19:27:33 DEBUG "value": [ 19:27:33 DEBUG "data": { 19:27:33 DEBUG 19:27:33 INFO [http_cmd]: http status code OK 19:27:33 DEBUG "resultType": "vector", 19:27:33 DEBUG 19:27:33 DEBUG --- status code --- 19:27:33 DEBUG "result": [ 19:27:33 DEBUG 19:27:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:33 DEBUG { 19:27:33 DEBUG [...] 19:27:33 DEBUG --- http response --- 19:27:33 DEBUG "metric": { 19:27:33 DEBUG 19:27:34 DEBUG "pod": "ds-idrepo-1" 19:27:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:27:34 DEBUG 19:27:34 DEBUG }, 19:27:34 DEBUG ----- output ----- 19:27:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205654" 19:27:34 DEBUG 19:27:34 DEBUG "value": [ 19:27:34 DEBUG { 19:27:34 DEBUG "instance": "10.142.0.43:9100" 19:27:34 DEBUG 19:27:34 DEBUG "status": "success", 19:27:34 DEBUG }, 19:27:34 DEBUG 19:27:34 DEBUG "data": { 19:27:34 DEBUG "value": [ 19:27:34 DEBUG 19:27:34 DEBUG "resultType": "vector", 19:27:34 DEBUG 1713205653, 19:27:34 DEBUG [...] 19:27:34 DEBUG "result": [ 19:27:34 DEBUG "0" 19:27:34 DEBUG 19:27:34 DEBUG { 19:27:34 DEBUG ] 19:27:34 DEBUG 19:27:34 DEBUG "metric": { 19:27:34 DEBUG } 19:27:34 DEBUG 19:27:34 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:27:34 DEBUG ] 19:27:34 DEBUG "pod": "ds-idrepo-0" 19:27:34 DEBUG }, 19:27:34 DEBUG } 19:27:34 DEBUG }, 19:27:34 DEBUG "value": [ 19:27:34 DEBUG } 19:27:34 DEBUG "value": [ 19:27:34 DEBUG 19:27:34 DEBUG 1713205653, 19:27:34 DEBUG 19:27:34 DEBUG "0" 19:27:34 DEBUG 19:27:34 DEBUG ] 19:27:34 DEBUG [...] 19:27:34 DEBUG } 19:27:34 DEBUG 19:27:34 DEBUG ] 19:27:34 DEBUG 19:27:34 DEBUG } 19:27:34 DEBUG 19:27:34 DEBUG } 19:27:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:27:34 DEBUG }, 19:27:34 DEBUG "value": [ 19:27:34 DEBUG 1713205653, 19:27:34 DEBUG "2425147.196557398" 19:27:34 DEBUG ] 19:27:34 DEBUG } 19:27:34 DEBUG ] 19:27:34 DEBUG } 19:27:34 DEBUG } 19:27:34 INFO [http_cmd]: http status code OK 19:27:34 DEBUG --- status code --- 19:27:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:34 DEBUG --- http response --- 19:27:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:27:34 DEBUG ----- output ----- 19:27:34 DEBUG { 19:27:34 DEBUG "status": "success", 19:27:34 DEBUG "data": { 19:27:34 DEBUG "resultType": "vector", 19:27:34 DEBUG "result": [ 19:27:34 DEBUG { 19:27:34 DEBUG "metric": { 19:27:34 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:27:34 DEBUG }, 19:27:34 DEBUG "value": [ 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG [...] 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG "pod": "ds-cts-2" 19:27:34 DEBUG }, 19:27:34 DEBUG "value": [ 19:27:34 DEBUG 1713205654, 19:27:34 DEBUG "50566.18147190858" 19:27:34 DEBUG ] 19:27:34 DEBUG } 19:27:34 DEBUG ] 19:27:34 DEBUG } 19:27:34 DEBUG } 19:27:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205654" 19:27:34 INFO [http_cmd]: http status code OK 19:27:34 DEBUG --- status code --- 19:27:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:34 DEBUG --- http response --- 19:27:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:34 DEBUG ----- output ----- 19:27:34 DEBUG { 19:27:34 DEBUG "status": "success", 19:27:34 DEBUG "data": { 19:27:34 DEBUG "resultType": "vector", 19:27:34 DEBUG "result": [ 19:27:34 DEBUG { 19:27:34 DEBUG "metric": { 19:27:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:27:34 DEBUG }, 19:27:34 DEBUG "value": [ 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG [...] 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:27:34 DEBUG }, 19:27:34 DEBUG "value": [ 19:27:34 DEBUG 1713205654, 19:27:34 DEBUG "90257.5" 19:27:34 DEBUG ] 19:27:34 DEBUG } 19:27:34 DEBUG ] 19:27:34 DEBUG } 19:27:34 DEBUG } 19:27:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205654" 19:27:34 INFO [http_cmd]: http status code OK 19:27:34 DEBUG --- status code --- 19:27:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:34 DEBUG --- http response --- 19:27:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:27:34 DEBUG ----- output ----- 19:27:34 DEBUG { 19:27:34 DEBUG "status": "success", 19:27:34 DEBUG "data": { 19:27:34 DEBUG "resultType": "vector", 19:27:34 DEBUG "result": [ 19:27:34 DEBUG { 19:27:34 DEBUG "metric": { 19:27:34 DEBUG "pod": "ds-idrepo-1" 19:27:34 DEBUG }, 19:27:34 DEBUG "value": [ 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG [...] 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG "pod": "ds-idrepo-0" 19:27:34 DEBUG }, 19:27:34 DEBUG "value": [ 19:27:34 DEBUG 1713205654, 19:27:34 DEBUG "0" 19:27:34 DEBUG ] 19:27:34 DEBUG } 19:27:34 DEBUG ] 19:27:34 DEBUG } 19:27:34 DEBUG } 19:27:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205654" 19:27:34 INFO [http_cmd]: http status code OK 19:27:34 DEBUG --- status code --- 19:27:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:27:34 DEBUG --- http response --- 19:27:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:27:34 DEBUG ----- output ----- 19:27:34 DEBUG { 19:27:34 DEBUG "status": "success", 19:27:34 DEBUG "data": { 19:27:34 DEBUG "resultType": "vector", 19:27:34 DEBUG "result": [ 19:27:34 DEBUG { 19:27:34 DEBUG "metric": { 19:27:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:27:34 DEBUG }, 19:27:34 DEBUG "value": [ 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG [...] 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG 19:27:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:27:34 DEBUG }, 19:27:34 DEBUG "value": [ 19:27:34 DEBUG 1713205654, 19:27:34 DEBUG "0" 19:27:34 DEBUG ] 19:27:34 DEBUG } 19:27:34 DEBUG ] 19:27:34 DEBUG } 19:27:34 DEBUG } 19:27:35 INFO 19:27:35 INFO [loop_until]: kubectl --namespace=xlou top pods 19:27:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:27:35 INFO [loop_until]: OK (rc = 0) 19:27:35 DEBUG --- stdout --- 19:27:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 1063m 8774Mi am-7d597cf7d6-8vr8f 1027m 8779Mi am-7d597cf7d6-9wz9h 1343m 8650Mi ds-cts-0 3368m 8394Mi ds-cts-1 2416m 8447Mi ds-cts-2 2359m 8599Mi ds-idrepo-0 1286m 4968Mi ds-idrepo-1 453m 4342Mi ds-idrepo-2 461m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 4m 1222Mi lodemon-7c47b75958-dgfm9 1m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1078m 656Mi 19:27:35 DEBUG --- stderr --- 19:27:35 DEBUG 19:27:36 INFO 19:27:36 INFO [loop_until]: kubectl --namespace=xlou top node 19:27:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:27:36 INFO [loop_until]: OK (rc = 0) 19:27:36 DEBUG --- stdout --- 19:27:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5093Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2329m 7% 10698Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1104m 3% 10892Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 94m 0% 3084Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 510m 1% 7096Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 3718m 12% 12699Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2979Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1164m 3% 10329Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 447m 1% 6220Mi 5% gke-xlou-cdm-ds-0f601751-j488 2395m 8% 10158Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2452Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 1526m 5% 10339Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 1353m 4% 7090Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 1094m 6% 4599Mi 7% 19:27:36 DEBUG --- stderr --- 19:27:36 DEBUG 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713205713" 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205713" 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205713" 19:28:33 INFO [http_cmd]: http status code OK 19:28:33 DEBUG --- status code --- 19:28:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:33 DEBUG --- http response --- 19:28:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:33 DEBUG ----- output ----- 19:28:33 DEBUG { 19:28:33 DEBUG "status": "success", 19:28:33 DEBUG "data": { 19:28:33 DEBUG "resultType": "vector", 19:28:33 DEBUG "result": [ 19:28:33 DEBUG { 19:28:33 DEBUG "metric": { 19:28:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG [...] 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 1713205713, 19:28:33 DEBUG "1.0306677327867908" 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG } 19:28:33 INFO [http_cmd]: http status code OK 19:28:33 DEBUG --- status code --- 19:28:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:33 DEBUG --- http response --- 19:28:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:33 DEBUG ----- output ----- 19:28:33 DEBUG { 19:28:33 DEBUG "status": "success", 19:28:33 DEBUG "data": { 19:28:33 DEBUG "resultType": "vector", 19:28:33 DEBUG "result": [ 19:28:33 DEBUG { 19:28:33 DEBUG "metric": { 19:28:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG [...] 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 1713205713, 19:28:33 DEBUG "1104.7666666666667" 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG } 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713205713" 19:28:33 INFO [http_cmd]: http status code OK 19:28:33 DEBUG --- status code --- 19:28:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:33 DEBUG --- http response --- 19:28:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205713" 19:28:33 ERROR [http_cmd]: ERROR 19:28:33 ERROR --- status code --- 19:28:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:28:33 ERROR --- http response --- 19:28:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:28:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:28:33 INFO [http_cmd]: http status code OK 19:28:33 DEBUG --- status code --- 19:28:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:33 DEBUG --- http response --- 19:28:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:33 DEBUG ----- output ----- 19:28:33 DEBUG { 19:28:33 DEBUG "status": "success", 19:28:33 DEBUG "data": { 19:28:33 DEBUG "resultType": "vector", 19:28:33 DEBUG "result": [ 19:28:33 DEBUG { 19:28:33 DEBUG "metric": { 19:28:33 DEBUG "instance": "10.142.0.107:9100" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG [...] 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG "instance": "10.142.0.43:9100" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 1713205713, 19:28:33 DEBUG "0.00040000000000001514" 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG } 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205713" 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205713" 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205713" 19:28:33 INFO [http_cmd]: http status code OK 19:28:33 INFO [http_cmd]: http status code OK 19:28:33 DEBUG --- status code --- 19:28:33 DEBUG --- status code --- 19:28:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:33 DEBUG --- http response --- 19:28:33 DEBUG --- http response --- 19:28:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:33 DEBUG ----- output ----- 19:28:33 DEBUG { 19:28:33 DEBUG "status": "success", 19:28:33 DEBUG "data": { 19:28:33 DEBUG "resultType": "vector", 19:28:33 DEBUG "result": [ 19:28:33 DEBUG { 19:28:33 DEBUG "metric": { 19:28:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG [...] 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 1713205713, 19:28:33 DEBUG "0" 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG } 19:28:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:33 DEBUG ----- output ----- 19:28:33 DEBUG { 19:28:33 DEBUG "status": "success", 19:28:33 DEBUG "data": { 19:28:33 DEBUG "resultType": "vector", 19:28:33 DEBUG "result": [ 19:28:33 DEBUG { 19:28:33 DEBUG "metric": { 19:28:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG [...] 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 1713205713, 19:28:33 DEBUG "0" 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG } 19:28:33 INFO [http_cmd]: http status code OK 19:28:33 DEBUG --- status code --- 19:28:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:33 DEBUG --- http response --- 19:28:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:28:33 DEBUG ----- output ----- 19:28:33 DEBUG { 19:28:33 DEBUG "status": "success", 19:28:33 DEBUG "data": { 19:28:33 DEBUG "resultType": "vector", 19:28:33 DEBUG "result": [ 19:28:33 DEBUG { 19:28:33 DEBUG "metric": { 19:28:33 DEBUG "pod": "ds-idrepo-1" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG [...] 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG "pod": "ds-idrepo-0" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 1713205713, 19:28:33 DEBUG "0" 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG } 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205713" 19:28:33 INFO [http_cmd]: http status code OK 19:28:33 DEBUG --- status code --- 19:28:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:33 DEBUG --- http response --- 19:28:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:33 DEBUG ----- output ----- 19:28:33 DEBUG { 19:28:33 DEBUG "status": "success", 19:28:33 DEBUG "data": { 19:28:33 DEBUG "resultType": "vector", 19:28:33 DEBUG "result": [ 19:28:33 DEBUG { 19:28:33 DEBUG "metric": { 19:28:33 DEBUG "pod": "ds-idrepo-1" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG [...] 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG "pod": "ds-idrepo-0" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 1713205713, 19:28:33 DEBUG "0" 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG } 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205713" 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205713" 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713205713" 19:28:33 INFO [http_cmd]: http status code OK 19:28:33 DEBUG --- status code --- 19:28:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:33 DEBUG --- http response --- 19:28:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:33 DEBUG ----- output ----- 19:28:33 DEBUG { 19:28:33 DEBUG "status": "success", 19:28:33 DEBUG "data": { 19:28:33 DEBUG "resultType": "vector", 19:28:33 DEBUG "result": [ 19:28:33 DEBUG { 19:28:33 DEBUG "metric": { 19:28:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG [...] 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 1713205713, 19:28:33 DEBUG "0" 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG } 19:28:33 INFO [http_cmd]: http status code OK 19:28:33 DEBUG --- status code --- 19:28:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:33 DEBUG --- http response --- 19:28:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:33 DEBUG ----- output ----- 19:28:33 DEBUG { 19:28:33 DEBUG "status": "success", 19:28:33 DEBUG "data": { 19:28:33 DEBUG "resultType": "vector", 19:28:33 DEBUG "result": [ 19:28:33 DEBUG { 19:28:33 DEBUG "metric": { 19:28:33 DEBUG "pod": "ds-idrepo-1" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG [...] 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG "pod": "ds-idrepo-0" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 1713205713, 19:28:33 DEBUG "0" 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG } 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713205713" 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205713" 19:28:33 INFO [http_cmd]: http status code OK 19:28:33 DEBUG --- status code --- 19:28:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:33 DEBUG --- http response --- 19:28:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:33 DEBUG ----- output ----- 19:28:33 DEBUG { 19:28:33 DEBUG "status": "success", 19:28:33 DEBUG "data": { 19:28:33 DEBUG "resultType": "vector", 19:28:33 DEBUG "result": [ 19:28:33 DEBUG { 19:28:33 DEBUG "metric": { 19:28:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG [...] 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 1713205713, 19:28:33 DEBUG "0.078" 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG } 19:28:33 INFO [http_cmd]: http status code OK 19:28:33 DEBUG --- status code --- 19:28:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:33 DEBUG --- http response --- 19:28:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:33 DEBUG ----- output ----- 19:28:33 DEBUG { 19:28:33 DEBUG "status": "success", 19:28:33 DEBUG "data": { 19:28:33 DEBUG "resultType": "vector", 19:28:33 DEBUG "result": [ 19:28:33 DEBUG { 19:28:33 DEBUG "metric": { 19:28:33 DEBUG "instance": "10.142.0.107:9100" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG [...] 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG "instance": "10.142.0.43:9100" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 1713205713, 19:28:33 DEBUG "13791456256" 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG } 19:28:33 INFO [http_cmd]: http status code OK 19:28:33 DEBUG --- status code --- 19:28:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:33 DEBUG --- http response --- 19:28:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:33 DEBUG ----- output ----- 19:28:33 DEBUG { 19:28:33 DEBUG "status": "success", 19:28:33 DEBUG "data": { 19:28:33 DEBUG "resultType": "vector", 19:28:33 DEBUG "result": [ 19:28:33 DEBUG { 19:28:33 DEBUG "metric": { 19:28:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG [...] 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG 19:28:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:28:33 DEBUG }, 19:28:33 DEBUG "value": [ 19:28:33 DEBUG 1713205713, 19:28:33 DEBUG "1402454016" 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG ] 19:28:33 DEBUG } 19:28:33 DEBUG } 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205713" 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205713" 19:28:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205713" 19:28:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205714" 19:28:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713205714" 19:28:34 INFO [http_cmd]: http status code OK 19:28:34 DEBUG --- status code --- 19:28:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:34 DEBUG --- http response --- 19:28:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:28:34 INFO [http_cmd]: http status code OK 19:28:34 INFO [http_cmd]: http status code OK 19:28:34 DEBUG ----- output ----- 19:28:34 DEBUG --- status code --- 19:28:34 DEBUG --- status code --- 19:28:34 DEBUG { 19:28:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:34 DEBUG "status": "success", 19:28:34 DEBUG --- http response --- 19:28:34 DEBUG --- http response --- 19:28:34 DEBUG "data": { 19:28:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:34 DEBUG "resultType": "vector", 19:28:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:34 INFO [http_cmd]: http status code OK 19:28:34 DEBUG ----- output ----- 19:28:34 INFO [http_cmd]: http status code OK 19:28:34 DEBUG "result": [ 19:28:34 DEBUG ----- output ----- 19:28:34 DEBUG --- status code --- 19:28:34 DEBUG { 19:28:34 DEBUG --- status code --- 19:28:34 DEBUG { 19:28:34 DEBUG { 19:28:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:34 DEBUG "status": "success", 19:28:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:34 DEBUG "metric": { 19:28:34 DEBUG "status": "success", 19:28:34 DEBUG --- http response --- 19:28:34 DEBUG "data": { 19:28:34 DEBUG --- http response --- 19:28:34 DEBUG "pod": "ds-idrepo-1" 19:28:34 DEBUG "data": { 19:28:34 DEBUG "resultType": "vector", 19:28:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:28:34 DEBUG }, 19:28:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:34 DEBUG "resultType": "vector", 19:28:34 DEBUG "result": [ 19:28:34 DEBUG ----- output ----- 19:28:34 DEBUG "value": [ 19:28:34 DEBUG ----- output ----- 19:28:34 DEBUG "result": [ 19:28:34 DEBUG { 19:28:34 DEBUG { 19:28:34 DEBUG 19:28:34 DEBUG { 19:28:34 DEBUG { 19:28:34 DEBUG "metric": { 19:28:34 DEBUG "status": "success", 19:28:34 DEBUG 19:28:34 DEBUG "status": "success", 19:28:34 DEBUG "metric": { 19:28:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:28:34 DEBUG "data": { 19:28:34 DEBUG 19:28:34 DEBUG "data": { 19:28:34 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:28:34 DEBUG }, 19:28:34 DEBUG "resultType": "vector", 19:28:34 DEBUG [...] 19:28:34 DEBUG "resultType": "vector", 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG "result": [ 19:28:34 DEBUG 19:28:34 DEBUG "result": [ 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 19:28:34 DEBUG { 19:28:34 DEBUG 19:28:34 DEBUG { 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG "metric": { 19:28:34 DEBUG 19:28:34 DEBUG "metric": { 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG "pod": "ds-idrepo-1" 19:28:34 DEBUG "pod": "ds-cts-2" 19:28:34 DEBUG "instance": "10.142.0.107:9100" 19:28:34 DEBUG 19:28:34 DEBUG [...] 19:28:34 DEBUG }, 19:28:34 DEBUG }, 19:28:34 DEBUG }, 19:28:34 DEBUG [...] 19:28:34 DEBUG 19:28:34 DEBUG "value": [ 19:28:34 DEBUG "value": [ 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 1713205713, 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG "0" 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:28:34 DEBUG 19:28:34 DEBUG ] 19:28:34 DEBUG 19:28:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:28:34 DEBUG }, 19:28:34 DEBUG [...] 19:28:34 DEBUG } 19:28:34 DEBUG [...] 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 19:28:34 DEBUG ] 19:28:34 DEBUG 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 1713205713, 19:28:34 DEBUG 19:28:34 DEBUG } 19:28:34 DEBUG 19:28:34 DEBUG 1713205713, 19:28:34 DEBUG "0" 19:28:34 DEBUG 19:28:34 DEBUG } 19:28:34 DEBUG 19:28:34 DEBUG "1.321639182080542" 19:28:34 DEBUG ] 19:28:34 DEBUG "pod": "ds-cts-2" 19:28:34 DEBUG "instance": "10.142.0.43:9100" 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG }, 19:28:34 DEBUG }, 19:28:34 DEBUG } 19:28:34 DEBUG ] 19:28:34 DEBUG "value": [ 19:28:34 DEBUG "value": [ 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG 1713205714, 19:28:34 DEBUG 1713205714, 19:28:34 DEBUG } 19:28:34 DEBUG } 19:28:34 DEBUG "0" 19:28:34 DEBUG "6616.466666666666" 19:28:34 DEBUG } 19:28:34 DEBUG ] 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG } 19:28:34 DEBUG ] 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG } 19:28:34 DEBUG } 19:28:34 DEBUG } 19:28:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205714" 19:28:34 INFO [http_cmd]: http status code OK 19:28:34 DEBUG --- status code --- 19:28:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:34 DEBUG --- http response --- 19:28:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:34 DEBUG ----- output ----- 19:28:34 DEBUG { 19:28:34 DEBUG "status": "success", 19:28:34 DEBUG "data": { 19:28:34 DEBUG "resultType": "vector", 19:28:34 DEBUG "result": [ 19:28:34 DEBUG { 19:28:34 DEBUG "metric": { 19:28:34 DEBUG "instance": "10.142.0.107:9100" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG [...] 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG "instance": "10.142.0.43:9100" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 1713205714, 19:28:34 DEBUG "6169.466666666666" 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG } 19:28:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205714" 19:28:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205714" 19:28:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205714" 19:28:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205714" 19:28:34 INFO [http_cmd]: http status code OK 19:28:34 DEBUG --- status code --- 19:28:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:34 DEBUG --- http response --- 19:28:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:34 DEBUG ----- output ----- 19:28:34 DEBUG { 19:28:34 DEBUG "status": "success", 19:28:34 DEBUG "data": { 19:28:34 DEBUG "resultType": "vector", 19:28:34 DEBUG "result": [ 19:28:34 DEBUG { 19:28:34 DEBUG "metric": { 19:28:34 DEBUG "instance": "10.142.0.107:9100" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG [...] 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG "instance": "10.142.0.43:9100" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 1713205714, 19:28:34 DEBUG "0.00033333333333333365" 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG } 19:28:34 INFO [http_cmd]: http status code OK 19:28:34 DEBUG --- status code --- 19:28:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:34 DEBUG --- http response --- 19:28:34 INFO [http_cmd]: http status code OK 19:28:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:34 DEBUG --- status code --- 19:28:34 DEBUG ----- output ----- 19:28:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:34 DEBUG { 19:28:34 DEBUG --- http response --- 19:28:34 DEBUG "status": "success", 19:28:34 DEBUG "data": { 19:28:34 DEBUG "resultType": "vector", 19:28:34 DEBUG "result": [ 19:28:34 DEBUG { 19:28:34 DEBUG "metric": { 19:28:34 DEBUG "pod": "ds-idrepo-1" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG [...] 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG "pod": "ds-idrepo-0" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 1713205714, 19:28:34 DEBUG "0" 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG } 19:28:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:34 DEBUG ----- output ----- 19:28:34 DEBUG { 19:28:34 DEBUG "status": "success", 19:28:34 DEBUG "data": { 19:28:34 DEBUG "resultType": "vector", 19:28:34 DEBUG "result": [ 19:28:34 DEBUG { 19:28:34 DEBUG "metric": { 19:28:34 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG [...] 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 1713205714, 19:28:34 DEBUG "1887124.9521022746" 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG } 19:28:34 INFO [http_cmd]: http status code OK 19:28:34 DEBUG --- status code --- 19:28:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:34 DEBUG --- http response --- 19:28:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:28:34 DEBUG ----- output ----- 19:28:34 DEBUG { 19:28:34 DEBUG "status": "success", 19:28:34 DEBUG "data": { 19:28:34 DEBUG "resultType": "vector", 19:28:34 DEBUG "result": [ 19:28:34 DEBUG { 19:28:34 DEBUG "metric": { 19:28:34 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG [...] 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 1713205714, 19:28:34 DEBUG "82042.70735360714" 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG } 19:28:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205714" 19:28:34 INFO [http_cmd]: http status code OK 19:28:34 DEBUG --- status code --- 19:28:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:34 DEBUG --- http response --- 19:28:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:28:34 DEBUG ----- output ----- 19:28:34 DEBUG { 19:28:34 DEBUG "status": "success", 19:28:34 DEBUG "data": { 19:28:34 DEBUG "resultType": "vector", 19:28:34 DEBUG "result": [ 19:28:34 DEBUG { 19:28:34 DEBUG "metric": { 19:28:34 DEBUG "pod": "ds-idrepo-1" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG [...] 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG "pod": "ds-idrepo-0" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 1713205714, 19:28:34 DEBUG "0" 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG } 19:28:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205714" 19:28:34 INFO [http_cmd]: http status code OK 19:28:34 DEBUG --- status code --- 19:28:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:34 DEBUG --- http response --- 19:28:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:34 DEBUG ----- output ----- 19:28:34 DEBUG { 19:28:34 DEBUG "status": "success", 19:28:34 DEBUG "data": { 19:28:34 DEBUG "resultType": "vector", 19:28:34 DEBUG "result": [ 19:28:34 DEBUG { 19:28:34 DEBUG "metric": { 19:28:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG [...] 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 1713205714, 19:28:34 DEBUG "25156.5" 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG } 19:28:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205714" 19:28:34 INFO [http_cmd]: http status code OK 19:28:34 DEBUG --- status code --- 19:28:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:28:34 DEBUG --- http response --- 19:28:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:28:34 DEBUG ----- output ----- 19:28:34 DEBUG { 19:28:34 DEBUG "status": "success", 19:28:34 DEBUG "data": { 19:28:34 DEBUG "resultType": "vector", 19:28:34 DEBUG "result": [ 19:28:34 DEBUG { 19:28:34 DEBUG "metric": { 19:28:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG [...] 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG 19:28:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:28:34 DEBUG }, 19:28:34 DEBUG "value": [ 19:28:34 DEBUG 1713205714, 19:28:34 DEBUG "0" 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG ] 19:28:34 DEBUG } 19:28:34 DEBUG } 19:28:35 INFO 19:28:35 INFO [loop_until]: kubectl --namespace=xlou top pods 19:28:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:28:35 INFO [loop_until]: OK (rc = 0) 19:28:35 DEBUG --- stdout --- 19:28:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 983m 8776Mi am-7d597cf7d6-8vr8f 982m 8779Mi am-7d597cf7d6-9wz9h 1216m 8648Mi ds-cts-0 3076m 8378Mi ds-cts-1 2229m 8451Mi ds-cts-2 2286m 8603Mi ds-idrepo-0 655m 4968Mi ds-idrepo-1 80m 4341Mi ds-idrepo-2 85m 4112Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 1025Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 998m 1546Mi 19:28:35 DEBUG --- stderr --- 19:28:35 DEBUG 19:28:36 INFO 19:28:36 INFO [loop_until]: kubectl --namespace=xlou top node 19:28:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:28:36 INFO [loop_until]: OK (rc = 0) 19:28:36 DEBUG --- stdout --- 19:28:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 5088Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2257m 7% 10704Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1052m 3% 10897Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 88m 0% 3070Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 131m 0% 7132Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 3492m 11% 12706Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2986Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 996m 3% 10330Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 127m 0% 6241Mi 5% gke-xlou-cdm-ds-0f601751-j488 2252m 7% 10146Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2455Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 1431m 4% 10336Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 575m 1% 7131Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 1051m 6% 5405Mi 9% 19:28:36 DEBUG --- stderr --- 19:28:36 DEBUG 19:29:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205773" 19:29:33 INFO [http_cmd]: http status code OK 19:29:33 DEBUG --- status code --- 19:29:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:33 DEBUG --- http response --- 19:29:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 19:29:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205773" 19:29:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713205773" 19:29:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205773" 19:29:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205773" 19:29:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205773" 19:29:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205773" 19:29:33 INFO [http_cmd]: http status code OK 19:29:33 DEBUG --- status code --- 19:29:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:33 DEBUG --- http response --- 19:29:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 19:29:33 DEBUG ----- output ----- 19:29:33 DEBUG { 19:29:33 DEBUG "status": "success", 19:29:33 DEBUG "data": { 19:29:33 DEBUG "resultType": "vector", 19:29:33 DEBUG "result": [ 19:29:33 DEBUG { 19:29:33 DEBUG "metric": { 19:29:33 DEBUG "pod": "ds-cts-1" 19:29:33 DEBUG }, 19:29:33 DEBUG "value": [ 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG [...] 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:29:33 DEBUG }, 19:29:33 DEBUG "value": [ 19:29:33 DEBUG 1713205773, 19:29:33 DEBUG "0" 19:29:33 DEBUG ] 19:29:33 DEBUG } 19:29:33 DEBUG ] 19:29:33 DEBUG } 19:29:33 DEBUG } 19:29:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713205773" 19:29:33 INFO [http_cmd]: http status code OK 19:29:33 INFO [http_cmd]: http status code OK 19:29:33 DEBUG --- status code --- 19:29:33 INFO [http_cmd]: http status code OK 19:29:33 INFO [http_cmd]: http status code OK 19:29:33 INFO [http_cmd]: http status code OK 19:29:33 DEBUG --- status code --- 19:29:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:33 DEBUG --- status code --- 19:29:33 DEBUG --- status code --- 19:29:33 DEBUG --- status code --- 19:29:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:33 DEBUG --- http response --- 19:29:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:33 DEBUG --- http response --- 19:29:33 DEBUG --- http response --- 19:29:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:29:33 DEBUG --- http response --- 19:29:33 DEBUG --- http response --- 19:29:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:29:33 DEBUG ----- output ----- 19:29:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:29:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:29:33 DEBUG ----- output ----- 19:29:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:29:33 DEBUG { 19:29:33 DEBUG ----- output ----- 19:29:33 DEBUG ----- output ----- 19:29:33 DEBUG { 19:29:33 DEBUG ----- output ----- 19:29:33 DEBUG "status": "success", 19:29:33 DEBUG { 19:29:33 DEBUG { 19:29:33 DEBUG "status": "success", 19:29:33 DEBUG { 19:29:33 DEBUG "data": { 19:29:33 DEBUG "status": "success", 19:29:33 DEBUG "status": "success", 19:29:33 DEBUG "data": { 19:29:33 DEBUG "status": "success", 19:29:33 DEBUG "resultType": "vector", 19:29:33 DEBUG "data": { 19:29:33 DEBUG "data": { 19:29:33 DEBUG "resultType": "vector", 19:29:33 DEBUG "data": { 19:29:33 DEBUG "result": [ 19:29:33 DEBUG "resultType": "vector", 19:29:33 DEBUG "resultType": "vector", 19:29:33 DEBUG "result": [ 19:29:33 DEBUG "resultType": "vector", 19:29:33 DEBUG { 19:29:33 DEBUG "result": [ 19:29:33 DEBUG "result": [ 19:29:33 DEBUG { 19:29:33 DEBUG "result": [ 19:29:33 DEBUG "metric": { 19:29:33 DEBUG { 19:29:33 DEBUG { 19:29:33 DEBUG "metric": { 19:29:33 DEBUG { 19:29:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:29:33 DEBUG "metric": { 19:29:33 DEBUG "metric": { 19:29:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:29:33 DEBUG "metric": { 19:29:33 DEBUG }, 19:29:33 DEBUG "pod": "ds-idrepo-1" 19:29:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:29:33 DEBUG }, 19:29:33 DEBUG "instance": "10.142.0.107:9100" 19:29:33 DEBUG "value": [ 19:29:33 DEBUG }, 19:29:33 DEBUG }, 19:29:33 DEBUG "value": [ 19:29:33 DEBUG }, 19:29:33 DEBUG 19:29:33 DEBUG "value": [ 19:29:33 DEBUG "value": [ 19:29:33 DEBUG 19:29:33 DEBUG "value": [ 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG [...] 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG [...] 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG [...] 19:29:33 DEBUG [...] 19:29:33 DEBUG 19:29:33 DEBUG [...] 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:29:33 DEBUG 19:29:33 DEBUG }, 19:29:33 DEBUG "pod": "ds-idrepo-0" 19:29:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:29:33 DEBUG }, 19:29:33 DEBUG "instance": "10.142.0.43:9100" 19:29:33 DEBUG "value": [ 19:29:33 DEBUG }, 19:29:33 DEBUG }, 19:29:33 DEBUG "value": [ 19:29:33 DEBUG }, 19:29:33 DEBUG 1713205773, 19:29:33 DEBUG "value": [ 19:29:33 DEBUG "value": [ 19:29:33 DEBUG 1713205773, 19:29:33 DEBUG "value": [ 19:29:33 DEBUG "0" 19:29:33 DEBUG 1713205773, 19:29:33 DEBUG 1713205773, 19:29:33 DEBUG "0" 19:29:33 DEBUG 1713205773, 19:29:33 DEBUG ] 19:29:33 DEBUG "0" 19:29:33 DEBUG "0.37310894282059764" 19:29:33 DEBUG ] 19:29:33 DEBUG "0.0003333333333330302" 19:29:33 DEBUG } 19:29:33 DEBUG ] 19:29:33 DEBUG ] 19:29:33 DEBUG } 19:29:33 DEBUG ] 19:29:33 DEBUG ] 19:29:33 DEBUG } 19:29:33 DEBUG } 19:29:33 DEBUG ] 19:29:33 DEBUG } 19:29:33 DEBUG } 19:29:33 DEBUG ] 19:29:33 DEBUG ] 19:29:33 DEBUG } 19:29:33 DEBUG ] 19:29:33 DEBUG } 19:29:33 DEBUG } 19:29:33 DEBUG } 19:29:33 DEBUG } 19:29:33 DEBUG } 19:29:33 DEBUG } 19:29:33 DEBUG } 19:29:33 DEBUG } 19:29:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205773" 19:29:33 ERROR [http_cmd]: ERROR 19:29:33 ERROR --- status code --- 19:29:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 19:29:33 ERROR --- http response --- 19:29:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 19:29:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 19:29:33 INFO [http_cmd]: http status code OK 19:29:33 DEBUG --- status code --- 19:29:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:33 DEBUG --- http response --- 19:29:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:29:33 DEBUG ----- output ----- 19:29:33 DEBUG { 19:29:33 DEBUG "status": "success", 19:29:33 DEBUG "data": { 19:29:33 DEBUG "resultType": "vector", 19:29:33 DEBUG "result": [ 19:29:33 DEBUG { 19:29:33 DEBUG "metric": { 19:29:33 DEBUG "pod": "ds-idrepo-1" 19:29:33 DEBUG }, 19:29:33 DEBUG "value": [ 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG [...] 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG 19:29:33 DEBUG "pod": "ds-idrepo-0" 19:29:33 DEBUG }, 19:29:33 DEBUG "value": [ 19:29:33 DEBUG 1713205773, 19:29:33 DEBUG "0" 19:29:33 DEBUG ] 19:29:33 DEBUG } 19:29:33 DEBUG ] 19:29:33 DEBUG } 19:29:33 DEBUG } 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205774" 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205774" 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205774" 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713205774" 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205774" 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG { 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "data": { 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG "result": [ 19:29:34 DEBUG { 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG "0" 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205774" 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713205774" 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 DEBUG --- status code --- 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG { 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG { 19:29:34 DEBUG { 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "data": { 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG { 19:29:34 DEBUG "data": { 19:29:34 DEBUG "data": { 19:29:34 DEBUG "result": [ 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG { 19:29:34 DEBUG "data": { 19:29:34 DEBUG "result": [ 19:29:34 DEBUG "result": [ 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG { 19:29:34 DEBUG { 19:29:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:29:34 DEBUG "result": [ 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "metric": { 19:29:34 DEBUG }, 19:29:34 DEBUG { 19:29:34 DEBUG "pod": "ds-idrepo-1" 19:29:34 DEBUG "pod": "ds-cts-1" 19:29:34 DEBUG "value": [ 19:29:34 DEBUG "metric": { 19:29:34 DEBUG }, 19:29:34 DEBUG }, 19:29:34 DEBUG 19:29:34 DEBUG "pod": "ds-idrepo-1" 19:29:34 DEBUG "value": [ 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG }, 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG }, 19:29:34 DEBUG 19:29:34 DEBUG "pod": "ds-cts-2" 19:29:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG }, 19:29:34 DEBUG }, 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG "pod": "ds-idrepo-0" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:29:34 DEBUG "0.078" 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:29:34 DEBUG "value": [ 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG ] 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG "0" 19:29:34 DEBUG { 19:29:34 DEBUG } 19:29:34 DEBUG { 19:29:34 DEBUG "0" 19:29:34 DEBUG "0" 19:29:34 DEBUG ] 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG ] 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG ] 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG "data": { 19:29:34 DEBUG } 19:29:34 DEBUG "data": { 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 DEBUG ] 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG } 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG ] 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG "result": [ 19:29:34 DEBUG "result": [ 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 DEBUG { 19:29:34 DEBUG { 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "instance": "10.142.0.107:9100" 19:29:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 19:29:34 DEBUG }, 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG "instance": "10.142.0.43:9100" 19:29:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 19:29:34 DEBUG }, 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG "13792171008" 19:29:34 DEBUG "1592774656" 19:29:34 DEBUG ] 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 DEBUG ] 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205774" 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713205774" 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205774" 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205774" 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG { 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "data": { 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG "result": [ 19:29:34 DEBUG { 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG "0" 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG { 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "data": { 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG "result": [ 19:29:34 DEBUG { 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "instance": "10.142.0.107:9100" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG "instance": "10.142.0.43:9100" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG "5175.766666666666" 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG { 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "data": { 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG "result": [ 19:29:34 DEBUG { 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "pod": "ds-idrepo-1" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG "pod": "ds-cts-2" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG "0" 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG { 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "data": { 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG "result": [ 19:29:34 DEBUG { 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "instance": "10.142.0.107:9100" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG "instance": "10.142.0.43:9100" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG "6076.4" 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713205774" 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205774" 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205774" 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205774" 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG { 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "data": { 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG "result": [ 19:29:34 DEBUG { 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "pod": "ds-idrepo-1" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG "pod": "ds-idrepo-0" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG "0" 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG { 19:29:34 DEBUG { 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "data": { 19:29:34 DEBUG "data": { 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG "result": [ 19:29:34 DEBUG "result": [ 19:29:34 DEBUG { 19:29:34 DEBUG { 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "pod": "ds-cts-1" 19:29:34 DEBUG "instance": "10.142.0.107:9100" 19:29:34 DEBUG }, 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 19:29:34 DEBUG "instance": "10.142.0.43:9100" 19:29:34 DEBUG }, 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG "0" 19:29:34 DEBUG "0" 19:29:34 DEBUG ] 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 DEBUG ] 19:29:34 DEBUG ] 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG { 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "data": { 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG "result": [ 19:29:34 DEBUG { 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "pod": "ds-cts-1" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG "0" 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205774" 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG { 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "data": { 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG "result": [ 19:29:34 DEBUG { 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "pod": "ds-idrepo-1" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG "pod": "ds-idrepo-0" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG "0" 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205774" 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG { 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "data": { 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG "result": [ 19:29:34 DEBUG { 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG "0" 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713205774" 19:29:34 INFO [http_cmd]: http status code OK 19:29:34 DEBUG --- status code --- 19:29:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 19:29:34 DEBUG --- http response --- 19:29:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 19:29:34 DEBUG ----- output ----- 19:29:34 DEBUG { 19:29:34 DEBUG "status": "success", 19:29:34 DEBUG "data": { 19:29:34 DEBUG "resultType": "vector", 19:29:34 DEBUG "result": [ 19:29:34 DEBUG { 19:29:34 DEBUG "metric": { 19:29:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG [...] 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG 19:29:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 19:29:34 DEBUG }, 19:29:34 DEBUG "value": [ 19:29:34 DEBUG 1713205774, 19:29:34 DEBUG "0" 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG ] 19:29:34 DEBUG } 19:29:34 DEBUG } 19:29:35 INFO 19:29:35 INFO [loop_until]: kubectl --namespace=xlou top pods 19:29:35 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:29:35 INFO [loop_until]: OK (rc = 0) 19:29:35 DEBUG --- stdout --- 19:29:35 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 7m 8763Mi am-7d597cf7d6-8vr8f 7m 8773Mi am-7d597cf7d6-9wz9h 7m 8639Mi ds-cts-0 13m 8379Mi ds-cts-1 21m 8420Mi ds-cts-2 18m 8573Mi ds-idrepo-0 16m 4969Mi ds-idrepo-1 9m 4341Mi ds-idrepo-2 10m 4113Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 3m 1025Mi idm-b85f9d978-rl76f 3m 1222Mi lodemon-7c47b75958-dgfm9 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1152m 1673Mi 19:29:35 DEBUG --- stderr --- 19:29:35 DEBUG 19:29:36 INFO 19:29:36 INFO [loop_until]: kubectl --namespace=xlou top node 19:29:36 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 19:29:36 INFO [loop_until]: OK (rc = 0) 19:29:36 DEBUG --- stdout --- 19:29:36 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 60m 0% 5088Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 83m 0% 10676Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 55m 0% 10887Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 92m 0% 3057Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 64m 0% 7135Mi 6% gke-xlou-cdm-default-pool-1b0769c8-wpwl 174m 0% 12695Mi 11% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2986Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 60m 0% 10324Mi 9% gke-xlou-cdm-ds-0f601751-hmm7 66m 0% 6254Mi 5% gke-xlou-cdm-ds-0f601751-j488 71m 0% 10128Mi 9% gke-xlou-cdm-ds-0f601751-mc5d 58m 0% 2444Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 83m 0% 10335Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 68m 0% 7117Mi 6% gke-xlou-cdm-frontend-a8771548-i82n 1277m 8% 5565Mi 9% 19:29:36 DEBUG --- stderr --- 19:29:36 DEBUG 19:29:42 INFO Finished: True 19:29:42 INFO Waiting for threads to register finish flag 19:30:36 INFO Done. Have a nice day! :) 127.0.0.1 - - [15/Apr/2024 19:30:36] "GET /monitoring/stop HTTP/1.1" 200 - 19:30:39 WARNING File /tmp/lodemon_data-20240415_184321/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6511', ...) 127.0.0.1 - - [15/Apr/2024 19:30:43] "GET /monitoring/process HTTP/1.1" 200 -