==================================================================================================== ========================================= Pod describe ========================================= ==================================================================================================== Name: lodemon-66fcb5cd8b-xh4df Namespace: xlou Priority: 0 Service Account: k8s-svc-acct-lodemon Node: gke-xlou-cdm-ds-0f601751-mc5d/10.142.0.43 Start Time: Sun, 14 Apr 2024 02:37:32 +0000 Labels: app=lodemon app.kubernetes.io/name=lodemon pod-template-hash=66fcb5cd8b skaffold.dev/run-id=5c16cdf1-c941-459a-bfd3-bd31b043de7d Annotations: Status: Running IP: 10.106.46.136 IPs: IP: 10.106.46.136 Controlled By: ReplicaSet/lodemon-66fcb5cd8b Containers: lodemon: Container ID: containerd://bc4786607572f7c5ca1b6b6adba9f550d2064c3134abb7c149a8841339c743b1 Image: gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable Image ID: gcr.io/engineeringpit/lodestar-images/lodestarbox@sha256:06c53901427b11990b989f0a16f422c2c89c055d3393605a0d2d532652d23afc Port: 8080/TCP Host Port: 0/TCP Command: python3 Args: /lodestar/scripts/lodemon_run.py -W default State: Running Started: Sun, 14 Apr 2024 02:37:32 +0000 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 2Gi Requests: cpu: 1 memory: 1Gi Liveness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Readiness: exec [cat /tmp/lodemon_alive] delay=20s timeout=1s period=10s #success=1 #failure=3 Environment: SKAFFOLD_PROFILE: medium Mounts: /lodestar/config/config.yaml from config (rw,path="config.yaml") /lodestar/config/deployments_to_monitor.yaml from deployments-config (rw,path="deployments_to_monitor.yaml") /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-rwxkv (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-config Optional: false deployments-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: lodemon-deployments Optional: false kube-api-access-rwxkv: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 13m default-scheduler Successfully assigned xlou/lodemon-66fcb5cd8b-xh4df to gke-xlou-cdm-ds-0f601751-mc5d Normal Pulling 13m kubelet Pulling image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" Normal Pulled 13m kubelet Successfully pulled image "gcr.io/engineeringpit/lodestar-images/lodestarbox:master-stable" in 232.69294ms (232.697559ms including waiting) Normal Created 13m kubelet Created container lodemon Normal Started 13m kubelet Started container lodemon ==================================================================================================== =========================================== Pod logs =========================================== ==================================================================================================== * Serving Flask app 'lodemon_run' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8080 * Running on http://10.106.46.136:8080 Press CTRL+C to quit 03:38:29 INFO 03:38:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:29 INFO [loop_until]: OK (rc = 0) 03:38:29 DEBUG --- stdout --- 03:38:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:29 DEBUG --- stderr --- 03:38:29 DEBUG 03:38:29 INFO 03:38:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:29 INFO [loop_until]: OK (rc = 0) 03:38:29 DEBUG --- stdout --- 03:38:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:29 DEBUG --- stderr --- 03:38:29 DEBUG 03:38:29 INFO 03:38:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:29 INFO [loop_until]: OK (rc = 0) 03:38:29 DEBUG --- stdout --- 03:38:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:29 DEBUG --- stderr --- 03:38:29 DEBUG 03:38:29 INFO 03:38:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:29 INFO [loop_until]: OK (rc = 0) 03:38:29 DEBUG --- stdout --- 03:38:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:29 DEBUG --- stderr --- 03:38:29 DEBUG 03:38:29 INFO 03:38:29 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:29 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:29 INFO [loop_until]: OK (rc = 0) 03:38:29 DEBUG --- stdout --- 03:38:29 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:29 DEBUG --- stderr --- 03:38:29 DEBUG 03:38:30 INFO 03:38:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:30 INFO [loop_until]: OK (rc = 0) 03:38:30 DEBUG --- stdout --- 03:38:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:30 DEBUG --- stderr --- 03:38:30 DEBUG 03:38:30 INFO 03:38:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:30 INFO [loop_until]: OK (rc = 0) 03:38:30 DEBUG --- stdout --- 03:38:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:30 DEBUG --- stderr --- 03:38:30 DEBUG 03:38:30 INFO 03:38:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:30 INFO [loop_until]: OK (rc = 0) 03:38:30 DEBUG --- stdout --- 03:38:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:30 DEBUG --- stderr --- 03:38:30 DEBUG 03:38:30 INFO 03:38:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:30 INFO [loop_until]: OK (rc = 0) 03:38:30 DEBUG --- stdout --- 03:38:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:30 DEBUG --- stderr --- 03:38:30 DEBUG 03:38:30 INFO 03:38:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:30 INFO [loop_until]: OK (rc = 0) 03:38:30 DEBUG --- stdout --- 03:38:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:30 DEBUG --- stderr --- 03:38:30 DEBUG 03:38:30 INFO 03:38:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:30 INFO [loop_until]: OK (rc = 0) 03:38:30 DEBUG --- stdout --- 03:38:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:30 DEBUG --- stderr --- 03:38:30 DEBUG 03:38:30 INFO 03:38:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:30 INFO [loop_until]: OK (rc = 0) 03:38:30 DEBUG --- stdout --- 03:38:30 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:30 DEBUG --- stderr --- 03:38:30 DEBUG 03:38:30 INFO 03:38:30 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:30 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:31 INFO [loop_until]: OK (rc = 0) 03:38:31 DEBUG --- stdout --- 03:38:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:31 DEBUG --- stderr --- 03:38:31 DEBUG 03:38:31 INFO 03:38:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:31 INFO [loop_until]: OK (rc = 0) 03:38:31 DEBUG --- stdout --- 03:38:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:31 DEBUG --- stderr --- 03:38:31 DEBUG 03:38:31 INFO 03:38:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:31 INFO [loop_until]: OK (rc = 0) 03:38:31 DEBUG --- stdout --- 03:38:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:31 DEBUG --- stderr --- 03:38:31 DEBUG 03:38:31 INFO 03:38:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:31 INFO [loop_until]: OK (rc = 0) 03:38:31 DEBUG --- stdout --- 03:38:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:31 DEBUG --- stderr --- 03:38:31 DEBUG 03:38:31 INFO 03:38:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:31 INFO [loop_until]: OK (rc = 0) 03:38:31 DEBUG --- stdout --- 03:38:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:31 DEBUG --- stderr --- 03:38:31 DEBUG 03:38:31 INFO 03:38:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:31 INFO [loop_until]: OK (rc = 0) 03:38:31 DEBUG --- stdout --- 03:38:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:31 DEBUG --- stderr --- 03:38:31 DEBUG 03:38:31 INFO 03:38:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:31 INFO [loop_until]: OK (rc = 0) 03:38:31 DEBUG --- stdout --- 03:38:31 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:31 DEBUG --- stderr --- 03:38:31 DEBUG 03:38:31 INFO 03:38:31 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:31 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:32 INFO [loop_until]: OK (rc = 0) 03:38:32 DEBUG --- stdout --- 03:38:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:32 DEBUG --- stderr --- 03:38:32 DEBUG 03:38:32 INFO 03:38:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:32 INFO [loop_until]: OK (rc = 0) 03:38:32 DEBUG --- stdout --- 03:38:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:32 DEBUG --- stderr --- 03:38:32 DEBUG 03:38:32 INFO 03:38:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:32 INFO [loop_until]: OK (rc = 0) 03:38:32 DEBUG --- stdout --- 03:38:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:32 DEBUG --- stderr --- 03:38:32 DEBUG 03:38:32 INFO 03:38:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:32 INFO [loop_until]: OK (rc = 0) 03:38:32 DEBUG --- stdout --- 03:38:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:32 DEBUG --- stderr --- 03:38:32 DEBUG 03:38:32 INFO 03:38:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:32 INFO [loop_until]: OK (rc = 0) 03:38:32 DEBUG --- stdout --- 03:38:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:32 DEBUG --- stderr --- 03:38:32 DEBUG 03:38:32 INFO 03:38:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:32 INFO [loop_until]: OK (rc = 0) 03:38:32 DEBUG --- stdout --- 03:38:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:32 DEBUG --- stderr --- 03:38:32 DEBUG 03:38:32 INFO 03:38:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:32 INFO [loop_until]: OK (rc = 0) 03:38:32 DEBUG --- stdout --- 03:38:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:32 DEBUG --- stderr --- 03:38:32 DEBUG 03:38:32 INFO 03:38:32 INFO [loop_until]: kubectl get services -o=jsonpath='{.items[?(@.metadata.labels.app=="kube-prometheus-stack-prometheus")]}' --all-namespaces 03:38:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:32 INFO [loop_until]: OK (rc = 0) 03:38:32 DEBUG --- stdout --- 03:38:32 DEBUG {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/neg":"{\"ingress\":true}","meta.helm.sh/release-name":"prometheus-operator","meta.helm.sh/release-namespace":"monitoring"},"creationTimestamp":"2023-05-27T02:35:19Z","labels":{"app":"kube-prometheus-stack-prometheus","app.kubernetes.io/instance":"prometheus-operator","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/part-of":"kube-prometheus-stack","app.kubernetes.io/version":"46.4.1","chart":"kube-prometheus-stack-46.4.1","heritage":"Helm","release":"prometheus-operator","self-monitor":"true"},"managedFields":[{"apiVersion":"v1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:meta.helm.sh/release-name":{},"f:meta.helm.sh/release-namespace":{}},"f:labels":{".":{},"f:app":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:chart":{},"f:heritage":{},"f:release":{},"f:self-monitor":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":9090,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}},"manager":"helm","operation":"Update","time":"2023-05-27T02:35:19Z"}],"name":"prometheus-operator-kube-p-prometheus","namespace":"monitoring","resourceVersion":"7148","uid":"eb1f35e8-cd91-4a12-8fe3-2e8bdf841da5"},"spec":{"clusterIP":"10.106.49.67","clusterIPs":["10.106.49.67"],"internalTrafficPolicy":"Cluster","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","ports":[{"name":"http-web","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app.kubernetes.io/name":"prometheus","prometheus":"prometheus-operator-kube-p-prometheus"},"sessionAffinity":"None","type":"ClusterIP"},"status":{"loadBalancer":{}}} 03:38:32 DEBUG --- stderr --- 03:38:32 DEBUG 03:38:32 INFO Initializing monitoring instance threads 03:38:32 DEBUG Monitoring instance thread list: [, , , , , , , , , , , , , , , , , , , , , , , , , , , , ] 03:38:32 INFO Starting instance threads 03:38:32 INFO 03:38:32 INFO Thread started 03:38:32 INFO [loop_until]: kubectl --namespace=xlou top node 03:38:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:32 INFO 03:38:32 INFO Thread started 03:38:32 INFO [loop_until]: kubectl --namespace=xlou top pods 03:38:32 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: http status code OK 03:38:32 INFO [http_cmd]: http status code OK 03:38:32 INFO [http_cmd]: http status code OK 03:38:32 INFO [http_cmd]: http status code OK 03:38:32 INFO [http_cmd]: http status code OK 03:38:32 INFO [http_cmd]: http status code OK 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062312" 03:38:32 DEBUG --- status code --- 03:38:32 INFO Thread started 03:38:32 INFO [http_cmd]: http status code OK 03:38:32 INFO [http_cmd]: http status code OK 03:38:32 DEBUG --- status code --- 03:38:32 DEBUG --- status code --- 03:38:32 DEBUG --- status code --- 03:38:32 DEBUG --- status code --- 03:38:32 DEBUG --- status code --- 03:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:32 INFO [http_cmd]: http status code OK 03:38:32 INFO [http_cmd]: http status code OK 03:38:32 INFO [http_cmd]: http status code OK 03:38:32 INFO [http_cmd]: http status code OK 03:38:32 INFO [http_cmd]: http status code OK 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062312" 03:38:32 DEBUG --- status code --- 03:38:32 INFO Thread started 03:38:32 DEBUG --- status code --- 03:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:32 DEBUG --- http response --- 03:38:32 DEBUG --- status code --- 03:38:32 INFO [http_cmd]: http status code OK 03:38:32 DEBUG --- status code --- 03:38:32 DEBUG --- status code --- 03:38:32 DEBUG --- status code --- 03:38:32 DEBUG --- status code --- 03:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:32 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:32 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062312" 03:38:32 INFO Thread started 03:38:32 DEBUG --- http response --- 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG --- status code --- 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG --- http response --- 03:38:33 INFO [http_cmd]: http status code OK 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:38:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062313" 03:38:33 INFO Thread started 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:33 DEBUG --- status code --- 03:38:33 DEBUG ----- output ----- 03:38:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062313" 03:38:33 INFO [http_cmd]: http status code OK 03:38:33 INFO Thread started 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG { 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:33 INFO [http_cmd]: http status code OK 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG { 03:38:33 INFO [loop_until]: OK (rc = 0) 03:38:33 DEBUG --- status code --- 03:38:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062313" 03:38:33 DEBUG { 03:38:33 INFO Thread started 03:38:33 DEBUG { 03:38:33 DEBUG { 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG { 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG --- status code --- 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG { 03:38:33 DEBUG { 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG --- stdout --- 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG "status": "success", 03:38:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062313" 03:38:33 DEBUG "status": "success", 03:38:33 INFO Thread started 03:38:33 DEBUG "data": { 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG { 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG { 03:38:33 DEBUG { 03:38:33 DEBUG { 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG { 03:38:33 INFO [http_cmd]: http status code OK 03:38:33 INFO [http_cmd]: http status code OK 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG "data": { 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8880Mi am-7d597cf7d6-8vr8f 6m 8483Mi am-7d597cf7d6-9wz9h 6m 8711Mi ds-cts-0 18m 7814Mi ds-cts-1 15m 8005Mi ds-cts-2 16m 8107Mi ds-idrepo-0 10m 4576Mi ds-idrepo-1 10m 4289Mi ds-idrepo-2 9m 4104Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 990Mi idm-b85f9d978-rl76f 4m 986Mi lodemon-66fcb5cd8b-xh4df 2m 54Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 1m 235Mi 03:38:33 DEBUG "data": { 03:38:33 DEBUG "data": { 03:38:33 DEBUG "data": { 03:38:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713062313" 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG "result": [ 03:38:33 DEBUG "data": { 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG { 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG --- status code --- 03:38:33 DEBUG --- status code --- 03:38:33 DEBUG "data": { 03:38:33 DEBUG "data": { 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG --- stderr --- 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG "resultType": "vector", 03:38:33 INFO [loop_until]: OK (rc = 0) 03:38:33 INFO Thread started 03:38:33 DEBUG { 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG "data": { 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG "data": { 03:38:33 DEBUG "data": { 03:38:33 DEBUG "data": { 03:38:33 DEBUG "data": { 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG "resultType": "vector", 03:38:33 INFO [http_cmd]: http status code OK 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG "result": [ 03:38:33 DEBUG { 03:38:33 DEBUG 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG "result": [ 03:38:33 DEBUG "result": [ 03:38:33 DEBUG "result": [ 03:38:33 DEBUG --- stdout --- 03:38:33 DEBUG "metric": { 03:38:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713062313" 03:38:33 DEBUG "result": [ 03:38:33 INFO Thread started 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG "data": { 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG "result": [ 03:38:33 DEBUG --- status code --- 03:38:33 DEBUG "result": [ 03:38:33 DEBUG { 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG { 03:38:33 DEBUG { 03:38:33 DEBUG { 03:38:33 DEBUG { 03:38:33 INFO [http_cmd]: http status code OK 03:38:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:38:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4809Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 72m 0% 10243Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 53m 0% 10944Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 87m 0% 3073Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 62m 0% 6323Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 169m 0% 11303Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2923Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 64m 0% 9931Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 63m 0% 5475Mi 4% gke-xlou-cdm-ds-0f601751-j488 69m 0% 9471Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 71m 0% 2236Mi 2% gke-xlou-cdm-ds-0f601751-nj6q 87m 0% 10252Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 69m 0% 6026Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 66m 0% 4039Mi 6% 03:38:33 DEBUG { 03:38:33 DEBUG "result": [ 03:38:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713062313" 03:38:33 DEBUG "resultType": "vector", 03:38:33 INFO Thread started 03:38:33 DEBUG "result": [ 03:38:33 DEBUG "result": [ 03:38:33 DEBUG "result": [ 03:38:33 DEBUG "result": [ 03:38:33 DEBUG { 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:38:33 DEBUG { 03:38:33 INFO [http_cmd]: http status code OK 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG { 03:38:33 DEBUG "metric": { 03:38:33 DEBUG "data": { 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG "metric": { 03:38:33 DEBUG "metric": { 03:38:33 DEBUG "metric": { 03:38:33 DEBUG --- status code --- 03:38:33 DEBUG }, 03:38:33 DEBUG --- stderr --- 03:38:33 DEBUG "metric": { 03:38:33 DEBUG { 03:38:33 DEBUG "result": [ 03:38:33 DEBUG { 03:38:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062313" 03:38:33 DEBUG { 03:38:33 INFO Thread started 03:38:33 DEBUG { 03:38:33 DEBUG { 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG "metric": { 03:38:33 DEBUG --- status code --- 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG "metric": { 03:38:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG "data": { 03:38:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:38:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:38:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG "value": [ 03:38:33 DEBUG 03:38:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:38:33 DEBUG "metric": { 03:38:33 DEBUG { 03:38:33 DEBUG "metric": { 03:38:33 DEBUG "metric": { 03:38:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062313" 03:38:33 DEBUG "metric": { 03:38:33 INFO Thread started 03:38:33 DEBUG "metric": { 03:38:33 DEBUG "data": { 03:38:33 DEBUG { 03:38:33 DEBUG { 03:38:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:38:33 INFO [http_cmd]: http status code OK 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG "instance": "10.142.0.107:9100" 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:38:33 DEBUG }, 03:38:33 DEBUG "result": [ 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG }, 03:38:33 DEBUG }, 03:38:33 DEBUG }, 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG 03:38:33 DEBUG }, 03:38:33 DEBUG "pod": "ds-idrepo-1" 03:38:33 DEBUG "metric": { 03:38:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:38:33 DEBUG "instance": "10.142.0.107:9100" 03:38:33 DEBUG "instance": "10.142.0.107:9100" 03:38:33 ERROR [http_cmd]: ERROR 03:38:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062313" 03:38:33 DEBUG "pod": "ds-idrepo-1" 03:38:33 INFO Thread started 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG }, 03:38:33 DEBUG --- status code --- 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG }, 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG "value": [ 03:38:33 DEBUG { 03:38:33 DEBUG "result": [ 03:38:33 DEBUG "value": [ 03:38:33 DEBUG "value": [ 03:38:33 DEBUG "value": [ 03:38:33 DEBUG 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:33 DEBUG "value": [ 03:38:33 DEBUG }, 03:38:33 DEBUG "pod": "ds-idrepo-1" 03:38:33 DEBUG }, 03:38:33 DEBUG }, 03:38:33 DEBUG }, 03:38:33 ERROR --- status code --- 03:38:33 DEBUG }, 03:38:33 INFO All threads has been started 127.0.0.1 - - [14/Apr/2024 03:38:33] "GET /monitoring/start HTTP/1.1" 200 - 03:38:33 DEBUG "result": [ 03:38:33 DEBUG "data": { 03:38:33 DEBUG "data": { 03:38:33 DEBUG "value": [ 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:38:33 DEBUG "value": [ 03:38:33 DEBUG { 03:38:33 DEBUG 03:38:33 DEBUG "metric": { 03:38:33 DEBUG { 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 INFO [http_cmd]: http status code OK 03:38:33 DEBUG 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG { 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG 03:38:33 DEBUG "value": [ 03:38:33 DEBUG "value": [ 03:38:33 DEBUG "value": [ 03:38:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:38:33 DEBUG "value": [ 03:38:33 DEBUG { 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "pod": "ds-idrepo-1" 03:38:33 DEBUG "metric": { 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 INFO [http_cmd]: http status code OK 03:38:33 DEBUG --- status code --- 03:38:33 DEBUG [...] 03:38:33 DEBUG "value": [ 03:38:33 DEBUG }, 03:38:33 DEBUG "data": { 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 ERROR --- http response --- 03:38:33 DEBUG 03:38:33 DEBUG "metric": { 03:38:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:38:33 DEBUG "result": [ 03:38:33 DEBUG 03:38:33 DEBUG { 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG }, 03:38:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG --- status code --- 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "value": [ 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "result": [ 03:38:33 DEBUG }, 03:38:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:38:33 DEBUG { 03:38:33 DEBUG 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG "data": { 03:38:33 DEBUG [...] 03:38:33 DEBUG [...] 03:38:33 DEBUG "value": [ 03:38:33 DEBUG }, 03:38:33 DEBUG [...] 03:38:33 DEBUG [...] 03:38:33 DEBUG 03:38:33 DEBUG [...] 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "result": [ 03:38:33 DEBUG [...] 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG { 03:38:33 DEBUG "value": [ 03:38:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:38:33 DEBUG "metric": { 03:38:33 DEBUG [...] 03:38:33 DEBUG "data": { 03:38:33 DEBUG { 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "value": [ 03:38:33 DEBUG 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG { 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG [...] 03:38:33 DEBUG [...] 03:38:33 DEBUG [...] 03:38:33 DEBUG [...] 03:38:33 DEBUG "metric": { 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "pod": "ds-idrepo-1" 03:38:33 INFO [http_cmd]: http status code OK 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG "result": [ 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 03:38:33 DEBUG 03:38:33 DEBUG { 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG }, 03:38:33 DEBUG --- status code --- 03:38:33 DEBUG "result": [ 03:38:33 DEBUG "data": { 03:38:33 DEBUG { 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:38:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:38:33 DEBUG 03:38:33 DEBUG }, 03:38:33 DEBUG [...] 03:38:33 DEBUG "metric": { 03:38:33 DEBUG "data": { 03:38:33 DEBUG [...] 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG }, 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "value": [ 03:38:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:38:33 DEBUG { 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG "metric": { 03:38:33 DEBUG "instance": "10.142.0.43:9100" 03:38:33 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 03:38:33 DEBUG [...] 03:38:33 DEBUG 03:38:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:38:33 DEBUG }, 03:38:33 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 03:38:33 DEBUG "value": [ 03:38:33 DEBUG 03:38:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG 03:38:33 DEBUG "pod": "login-ui-5fdf49fc66-2wnx7" 03:38:33 DEBUG 03:38:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "value": [ 03:38:33 DEBUG [...] 03:38:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:38:33 DEBUG 03:38:33 DEBUG --- http response --- 03:38:33 DEBUG "metric": { 03:38:33 DEBUG "result": [ 03:38:33 DEBUG "pod": "ds-idrepo-1" 03:38:33 DEBUG }, 03:38:33 DEBUG }, 03:38:33 DEBUG 03:38:33 DEBUG [...] 03:38:33 DEBUG }, 03:38:33 DEBUG "value": [ 03:38:33 DEBUG }, 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG 03:38:33 DEBUG }, 03:38:33 DEBUG "result": [ 03:38:33 DEBUG 03:38:33 DEBUG }, 03:38:33 DEBUG 03:38:33 DEBUG }, 03:38:33 DEBUG "instance": "10.142.0.43:9100" 03:38:33 DEBUG "pod": "ds-idrepo-0" 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG }, 03:38:33 DEBUG 03:38:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:38:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:38:33 DEBUG { 03:38:33 DEBUG }, 03:38:33 DEBUG "value": [ 03:38:33 DEBUG "value": [ 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "value": [ 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG "value": [ 03:38:33 DEBUG "0" 03:38:33 DEBUG 03:38:33 DEBUG "value": [ 03:38:33 DEBUG { 03:38:33 DEBUG 03:38:33 DEBUG "value": [ 03:38:33 DEBUG "instance": "10.142.0.43:9100" 03:38:33 DEBUG "value": [ 03:38:33 DEBUG }, 03:38:33 DEBUG }, 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "value": [ 03:38:33 DEBUG 03:38:33 DEBUG }, 03:38:33 DEBUG ----- output ----- 03:38:33 DEBUG "metric": { 03:38:33 DEBUG "value": [ 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG "343437312" 03:38:33 DEBUG "0.026875830964540007" 03:38:33 DEBUG ] 03:38:33 DEBUG "pod": "ds-idrepo-0" 03:38:33 DEBUG 03:38:33 DEBUG "metric": { 03:38:33 DEBUG "pod": "ds-idrepo-0" 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG }, 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG "value": [ 03:38:33 DEBUG "value": [ 03:38:33 DEBUG 03:38:33 DEBUG [...] 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG [...] 03:38:33 DEBUG 03:38:33 DEBUG "value": [ 03:38:33 DEBUG "pod": "ds-idrepo-1" 03:38:33 DEBUG 03:38:33 DEBUG "5545.966666666666" 03:38:33 DEBUG "0" 03:38:33 DEBUG "pod": "ds-idrepo-0" 03:38:33 DEBUG 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG ] 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG }, 03:38:33 DEBUG 03:38:33 DEBUG "instance": "10.142.0.107:9100" 03:38:33 DEBUG }, 03:38:33 DEBUG "179210.76987799746" 03:38:33 DEBUG "value": [ 03:38:33 DEBUG "0" 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG "0" 03:38:33 DEBUG 03:38:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:38:33 DEBUG { 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG }, 03:38:33 DEBUG 03:38:33 DEBUG ] 03:38:33 DEBUG ] 03:38:33 DEBUG }, 03:38:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:38:33 DEBUG "0.000016407236011780395" 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG "value": [ 03:38:33 DEBUG 03:38:33 DEBUG }, 03:38:33 DEBUG "value": [ 03:38:33 DEBUG ] 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG ] 03:38:33 DEBUG "4221.700000000001" 03:38:33 DEBUG 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG "0" 03:38:33 DEBUG "status": "success", 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "value": [ 03:38:33 DEBUG 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG "value": [ 03:38:33 DEBUG }, 03:38:33 DEBUG ] 03:38:33 DEBUG ] 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG [...] 03:38:33 DEBUG "value": [ 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG } 03:38:33 DEBUG "0" 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG 03:38:33 DEBUG }, 03:38:33 DEBUG ] 03:38:33 DEBUG ] 03:38:33 DEBUG "data": { 03:38:33 DEBUG "pod": "ds-idrepo-0" 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG [...] 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG "value": [ 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG "0" 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "0" 03:38:33 DEBUG ] 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG 03:38:33 DEBUG "value": [ 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG "resultType": "vector", 03:38:33 DEBUG }, 03:38:33 DEBUG [...] 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG "0" 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG ] 03:38:33 DEBUG ] 03:38:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:38:33 DEBUG 1713062312, 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG "result": [ 03:38:33 DEBUG "value": [ 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG "0" 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG }, 03:38:33 DEBUG "0" 03:38:33 DEBUG } 03:38:33 DEBUG { 03:38:33 DEBUG 1713062313, 03:38:33 DEBUG 03:38:33 DEBUG [...] 03:38:33 DEBUG 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG 03:38:33 DEBUG } 03:38:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG "value": [ 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG "metric": { 03:38:33 DEBUG "0" 03:38:33 DEBUG 03:38:33 DEBUG 03:38:33 DEBUG "pod": "ds-cts-1" 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG [...] 03:38:33 DEBUG } 03:38:33 DEBUG }, 03:38:33 DEBUG } 03:38:33 DEBUG 1713062313, 03:38:33 DEBUG } 03:38:33 DEBUG "instance": "10.142.0.107:9100" 03:38:33 DEBUG ] 03:38:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:38:33 DEBUG 03:38:33 DEBUG }, 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG 03:38:33 DEBUG "value": [ 03:38:33 DEBUG 1713062313, 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG }, 03:38:33 DEBUG } 03:38:33 DEBUG }, 03:38:33 DEBUG 03:38:33 DEBUG "pod": "ds-cts-1" 03:38:33 DEBUG "value": [ 03:38:33 DEBUG } 03:38:33 DEBUG 03:38:33 DEBUG "0" 03:38:33 DEBUG "1" 03:38:33 DEBUG } 03:38:33 DEBUG "value": [ 03:38:33 DEBUG ] 03:38:33 DEBUG "value": [ 03:38:33 DEBUG } 03:38:33 DEBUG }, 03:38:33 DEBUG 1713062313, 03:38:33 DEBUG } 03:38:33 DEBUG 03:38:33 DEBUG ] 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG 03:38:33 DEBUG } 03:38:33 DEBUG 1713062313, 03:38:33 DEBUG "value": [ 03:38:33 DEBUG "0" 03:38:33 DEBUG "instance": "10.142.0.43:9100" 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG 03:38:33 DEBUG } 03:38:33 DEBUG "0.015" 03:38:33 DEBUG 1713062313, 03:38:33 DEBUG ] 03:38:33 DEBUG }, 03:38:33 DEBUG ] 03:38:33 DEBUG ] 03:38:33 DEBUG 03:38:33 DEBUG ] 03:38:33 DEBUG "0" 03:38:33 DEBUG } 03:38:33 DEBUG "value": [ 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG [...] 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG ] 03:38:33 DEBUG 1713062313, 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG "12662907904" 03:38:33 DEBUG 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG } 03:38:33 DEBUG "instance": "10.142.0.43:9100" 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG }, 03:38:33 DEBUG } 03:38:33 DEBUG "value": [ 03:38:33 DEBUG } 03:38:33 DEBUG 1713062313, 03:38:33 DEBUG "0.00036666666666675953" 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG ] 03:38:33 DEBUG } 03:38:33 DEBUG } 03:39:33 INFO 03:39:33 INFO [loop_until]: kubectl --namespace=xlou top pods 03:39:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062373" 03:39:33 INFO [loop_until]: kubectl --namespace=xlou top node 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062373" 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG --- status code --- 03:39:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062373" 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 DEBUG --- status code --- 03:39:33 INFO [loop_until]: OK (rc = 0) 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG --- status code --- 03:39:33 ERROR [http_cmd]: ERROR 03:39:33 DEBUG --- status code --- 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG --- status code --- 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG --- status code --- 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG --- status code --- 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG --- stdout --- 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 ERROR --- status code --- 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5m 8879Mi am-7d597cf7d6-8vr8f 6m 8492Mi am-7d597cf7d6-9wz9h 5m 8711Mi ds-cts-0 477m 7941Mi ds-cts-1 432m 7976Mi ds-cts-2 374m 8107Mi ds-idrepo-0 423m 4576Mi ds-idrepo-1 482m 4416Mi ds-idrepo-2 472m 4104Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 991Mi idm-b85f9d978-rl76f 4m 988Mi lodemon-66fcb5cd8b-xh4df 2m 61Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 223m 384Mi 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG --- http response --- 03:39:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG --- http response --- 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG --- http response --- 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 INFO [http_cmd]: http status code OK 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG --- stderr --- 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:39:33 ERROR --- http response --- 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:39:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG --- status code --- 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG { 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:39:33 DEBUG { 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG { 03:39:33 DEBUG { 03:39:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:39:33 DEBUG { 03:39:33 DEBUG { 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG { 03:39:33 DEBUG { 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG --- http response --- 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:39:33 DEBUG { 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG { 03:39:33 DEBUG { 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG { 03:39:33 DEBUG { 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG { 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG { 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG { 03:39:33 DEBUG { 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "data": { 03:39:33 DEBUG { 03:39:33 DEBUG "data": { 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG { 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "data": { 03:39:33 DEBUG "data": { 03:39:33 DEBUG "data": { 03:39:33 DEBUG "data": { 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG "data": { 03:39:33 DEBUG "data": { 03:39:33 DEBUG { 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG { 03:39:33 DEBUG ----- output ----- 03:39:33 DEBUG "data": { 03:39:33 DEBUG { 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "data": { 03:39:33 DEBUG "data": { 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "data": { 03:39:33 DEBUG "data": { 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "data": { 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "data": { 03:39:33 DEBUG { 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG { 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG { 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "data": { 03:39:33 DEBUG "data": { 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "data": { 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "data": { 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "data": { 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "data": { 03:39:33 DEBUG "status": "success", 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "data": { 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "result": [ 03:39:33 DEBUG { 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG { 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "result": [ 03:39:33 DEBUG { 03:39:33 DEBUG { 03:39:33 DEBUG { 03:39:33 DEBUG { 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "data": { 03:39:33 DEBUG { 03:39:33 DEBUG { 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "data": { 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "data": { 03:39:33 DEBUG { 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "result": [ 03:39:33 DEBUG { 03:39:33 DEBUG { 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "metric": { 03:39:33 DEBUG { 03:39:33 DEBUG { 03:39:33 DEBUG "result": [ 03:39:33 DEBUG { 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "metric": { 03:39:33 DEBUG { 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "resultType": "vector", 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "result": [ 03:39:33 DEBUG { 03:39:33 DEBUG { 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:39:33 DEBUG { 03:39:33 DEBUG "pod": "ds-idrepo-1" 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "metric": { 03:39:33 DEBUG { 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "pod": "ds-cts-2" 03:39:33 DEBUG "instance": "10.142.0.107:9100" 03:39:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:39:33 DEBUG "pod": "ds-idrepo-1" 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "instance": "10.142.0.107:9100" 03:39:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:39:33 DEBUG { 03:39:33 DEBUG "result": [ 03:39:33 DEBUG { 03:39:33 DEBUG "result": [ 03:39:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:39:33 DEBUG { 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:39:33 DEBUG "pod": "ds-cts-2" 03:39:33 DEBUG }, 03:39:33 DEBUG "metric": { 03:39:33 DEBUG }, 03:39:33 DEBUG "instance": "10.142.0.107:9100" 03:39:33 DEBUG "pod": "ds-cts-2" 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "pod": "ds-idrepo-1" 03:39:33 DEBUG }, 03:39:33 DEBUG }, 03:39:33 DEBUG }, 03:39:33 DEBUG }, 03:39:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:39:33 DEBUG { 03:39:33 DEBUG }, 03:39:33 DEBUG }, 03:39:33 DEBUG "metric": { 03:39:33 DEBUG { 03:39:33 DEBUG "metric": { 03:39:33 DEBUG { 03:39:33 DEBUG }, 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:39:33 DEBUG "instance": "10.142.0.107:9100" 03:39:33 DEBUG }, 03:39:33 DEBUG }, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "pod": "ds-idrepo-1" 03:39:33 DEBUG "value": [ 03:39:33 DEBUG }, 03:39:33 DEBUG }, 03:39:33 DEBUG "instance": "10.142.0.107:9100" 03:39:33 DEBUG }, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "value": [ 03:39:33 DEBUG }, 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "pod": "ds-cts-2" 03:39:33 DEBUG "metric": { 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "pod": "ds-idrepo-1" 03:39:33 DEBUG }, 03:39:33 DEBUG }, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 03:39:33 DEBUG }, 03:39:33 DEBUG 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "value": [ 03:39:33 DEBUG }, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "pod": "ds-idrepo-1" 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG }, 03:39:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:39:33 DEBUG }, 03:39:33 DEBUG "pod": "ds-idrepo-1" 03:39:33 DEBUG 03:39:33 DEBUG }, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 INFO [loop_until]: OK (rc = 0) 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG }, 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG "value": [ 03:39:33 DEBUG }, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG }, 03:39:33 DEBUG 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG --- stdout --- 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4826Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 395m 1% 10287Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 56m 0% 10933Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 95m 0% 3060Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 555m 1% 6291Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 810m 2% 11259Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 66m 0% 2920Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 61m 0% 9923Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 417m 1% 5432Mi 4% gke-xlou-cdm-ds-0f601751-j488 401m 1% 9424Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 59m 0% 2214Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 86m 0% 10244Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 616m 2% 5987Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 297m 1% 4185Mi 7% 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG [...] 03:39:33 DEBUG [...] 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG --- stderr --- 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG [...] 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:39:33 DEBUG 03:39:33 DEBUG "pod": "ds-cts-1" 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG "pod": "ds-cts-1" 03:39:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:39:33 DEBUG 03:39:33 DEBUG }, 03:39:33 DEBUG "pod": "ds-idrepo-0" 03:39:33 DEBUG 03:39:33 DEBUG "instance": "10.142.0.43:9100" 03:39:33 DEBUG 03:39:33 DEBUG }, 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:39:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:39:33 DEBUG }, 03:39:33 DEBUG 03:39:33 DEBUG }, 03:39:33 DEBUG "instance": "10.142.0.43:9100" 03:39:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:39:33 DEBUG 03:39:33 DEBUG "instance": "10.142.0.43:9100" 03:39:33 DEBUG }, 03:39:33 DEBUG }, 03:39:33 DEBUG 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG 03:39:33 DEBUG }, 03:39:33 DEBUG 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG 03:39:33 DEBUG }, 03:39:33 DEBUG 03:39:33 DEBUG "pod": "ds-idrepo-0" 03:39:33 DEBUG "instance": "10.142.0.43:9100" 03:39:33 DEBUG }, 03:39:33 DEBUG }, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "pod": "ds-idrepo-0" 03:39:33 DEBUG "value": [ 03:39:33 DEBUG }, 03:39:33 DEBUG }, 03:39:33 DEBUG "instance": "10.142.0.43:9100" 03:39:33 DEBUG }, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:39:33 DEBUG }, 03:39:33 DEBUG "0" 03:39:33 DEBUG 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG 03:39:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:39:33 DEBUG }, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:39:33 DEBUG }, 03:39:33 DEBUG }, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG }, 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "value": [ 03:39:33 DEBUG }, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG }, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG ] 03:39:33 DEBUG "pod": "ds-idrepo-0" 03:39:33 DEBUG 03:39:33 DEBUG "12665859072" 03:39:33 DEBUG "0.2297599981158354" 03:39:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:39:33 DEBUG }, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG }, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG "0" 03:39:33 DEBUG "value": [ 03:39:33 DEBUG "0" 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG "0" 03:39:33 DEBUG "0" 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG } 03:39:33 DEBUG }, 03:39:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG "0.015" 03:39:33 DEBUG "value": [ 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG "0" 03:39:33 DEBUG "0" 03:39:33 DEBUG ] 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG ] 03:39:33 DEBUG "3074.9666666666667" 03:39:33 DEBUG "0.004125792418667822" 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG "5201.533333333334" 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG "0" 03:39:33 DEBUG ] 03:39:33 DEBUG "value": [ 03:39:33 DEBUG }, 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG "pod": "ds-idrepo-0" 03:39:33 DEBUG "19741.095801798307" 03:39:33 DEBUG ] 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG "343867392" 03:39:33 DEBUG "0" 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG "0" 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG "0.0010000000000000009" 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG "0" 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG "value": [ 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG "1" 03:39:33 DEBUG }, 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG "0.0008333333333335228" 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG "value": [ 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG "0" 03:39:33 DEBUG "0" 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG 1713062373, 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG "0" 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG ] 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:39:33 DEBUG } 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062433" 03:40:33 INFO 03:40:33 INFO [loop_until]: kubectl --namespace=xlou top node 03:40:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062433" 03:40:33 INFO 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [loop_until]: kubectl --namespace=xlou top pods 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 ERROR [http_cmd]: ERROR 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062433" 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 DEBUG --- status code --- 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 DEBUG --- status code --- 03:40:33 ERROR --- status code --- 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG --- status code --- 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG --- status code --- 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062433" 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713062433" 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 DEBUG --- status code --- 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062433" 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG --- status code --- 03:40:33 ERROR --- http response --- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:40:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:40:33 DEBUG --- http response --- 03:40:33 INFO [http_cmd]: http status code OK 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG --- status code --- 03:40:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:40:33 DEBUG { 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:40:33 DEBUG --- status code --- 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG { 03:40:33 DEBUG { 03:40:33 DEBUG { 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG { 03:40:33 DEBUG { 03:40:33 DEBUG { 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG { 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:40:33 DEBUG { 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG { 03:40:33 DEBUG { 03:40:33 DEBUG { 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG { 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG { 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "data": { 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG { 03:40:33 DEBUG "data": { 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG { 03:40:33 DEBUG --- http response --- 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:40:33 DEBUG "data": { 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "data": { 03:40:33 DEBUG "data": { 03:40:33 DEBUG "data": { 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:40:33 DEBUG "data": { 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG { 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:40:33 DEBUG { 03:40:33 DEBUG "data": { 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:40:33 DEBUG { 03:40:33 DEBUG "data": { 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "data": { 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "data": { 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG "result": [ 03:40:33 DEBUG { 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "data": { 03:40:33 DEBUG "data": { 03:40:33 DEBUG "result": [ 03:40:33 DEBUG { 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "data": { 03:40:33 DEBUG { 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG { 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "result": [ 03:40:33 DEBUG { 03:40:33 DEBUG "data": { 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "data": { 03:40:33 DEBUG { 03:40:33 DEBUG "data": { 03:40:33 DEBUG "result": [ 03:40:33 DEBUG { 03:40:33 DEBUG "data": { 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "result": [ 03:40:33 DEBUG { 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG { 03:40:33 DEBUG "metric": { 03:40:33 DEBUG { 03:40:33 DEBUG { 03:40:33 DEBUG { 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "pod": "ds-cts-2" 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG { 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG { 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG { 03:40:33 DEBUG "instance": "10.142.0.107:9100" 03:40:33 DEBUG { 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "data": { 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "data": { 03:40:33 DEBUG "result": [ 03:40:33 DEBUG }, 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "data": { 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "data": { 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "instance": "10.142.0.107:9100" 03:40:33 DEBUG }, 03:40:33 DEBUG "metric": { 03:40:33 DEBUG }, 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "pod": "ds-idrepo-1" 03:40:33 DEBUG { 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "data": { 03:40:33 DEBUG }, 03:40:33 DEBUG "instance": "10.142.0.107:9100" 03:40:33 DEBUG }, 03:40:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:40:33 DEBUG }, 03:40:33 DEBUG { 03:40:33 DEBUG "value": [ 03:40:33 DEBUG 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "pod": "ds-idrepo-1" 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "pod": "ds-cts-2" 03:40:33 DEBUG "metric": { 03:40:33 DEBUG 03:40:33 DEBUG }, 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "metric": { 03:40:33 DEBUG 03:40:33 DEBUG { 03:40:33 DEBUG }, 03:40:33 DEBUG { 03:40:33 DEBUG { 03:40:33 DEBUG 03:40:33 DEBUG }, 03:40:33 DEBUG "pod": "ds-idrepo-1" 03:40:33 DEBUG 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:40:33 DEBUG }, 03:40:33 DEBUG { 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "pod": "ds-idrepo-1" 03:40:33 DEBUG 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "metric": { 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG }, 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "data": { 03:40:33 DEBUG { 03:40:33 DEBUG "pod": "ds-idrepo-1" 03:40:33 DEBUG { 03:40:33 DEBUG { 03:40:33 DEBUG { 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "metric": { 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG }, 03:40:33 DEBUG [...] 03:40:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:40:33 DEBUG 03:40:33 DEBUG "pod": "ds-cts-2" 03:40:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:40:33 DEBUG "value": [ 03:40:33 DEBUG [...] 03:40:33 DEBUG "value": [ 03:40:33 DEBUG [...] 03:40:33 DEBUG 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG "metric": { 03:40:33 DEBUG }, 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "result": [ 03:40:33 DEBUG ----- output ----- 03:40:33 DEBUG { 03:40:33 DEBUG 03:40:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "value": [ 03:40:33 DEBUG 03:40:33 DEBUG }, 03:40:33 DEBUG 03:40:33 DEBUG }, 03:40:33 DEBUG }, 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "result": [ 03:40:33 DEBUG "pod": "ds-idrepo-1" 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:40:33 DEBUG "instance": "10.142.0.107:9100" 03:40:33 DEBUG "pod": "ds-idrepo-1" 03:40:33 DEBUG { 03:40:33 DEBUG { 03:40:33 DEBUG "metric": { 03:40:33 DEBUG 03:40:33 DEBUG }, 03:40:33 DEBUG [...] 03:40:33 DEBUG [...] 03:40:33 DEBUG [...] 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "value": [ 03:40:33 DEBUG 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "value": [ 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG { 03:40:33 DEBUG }, 03:40:33 DEBUG 03:40:33 DEBUG }, 03:40:33 DEBUG }, 03:40:33 DEBUG }, 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "status": "success", 03:40:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:40:33 DEBUG }, 03:40:33 DEBUG "value": [ 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG [...] 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG [...] 03:40:33 DEBUG "instance": "10.142.0.43:9100" 03:40:33 DEBUG "metric": { 03:40:33 DEBUG "value": [ 03:40:33 DEBUG 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:40:33 DEBUG "data": { 03:40:33 DEBUG "value": [ 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG [...] 03:40:33 DEBUG "instance": "10.142.0.43:9100" 03:40:33 DEBUG [...] 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "pod": "ds-cts-2" 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG }, 03:40:33 DEBUG "resultType": "vector", 03:40:33 DEBUG 03:40:33 DEBUG [...] 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG [...] 03:40:33 DEBUG }, 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 INFO [loop_until]: OK (rc = 0) 03:40:33 DEBUG }, 03:40:33 DEBUG }, 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG }, 03:40:33 DEBUG 03:40:33 DEBUG [...] 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "result": [ 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:40:33 DEBUG "instance": "10.142.0.43:9100" 03:40:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:40:33 DEBUG 03:40:33 DEBUG "value": [ 03:40:33 DEBUG [...] 03:40:33 DEBUG 03:40:33 DEBUG "pod": "ds-idrepo-0" 03:40:33 DEBUG }, 03:40:33 DEBUG 03:40:33 DEBUG --- stdout --- 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "value": [ 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "value": [ 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG { 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG [...] 03:40:33 DEBUG }, 03:40:33 DEBUG }, 03:40:33 DEBUG }, 03:40:33 DEBUG 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG 03:40:33 DEBUG [...] 03:40:33 DEBUG [...] 03:40:33 DEBUG "value": [ 03:40:33 DEBUG 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 67m 0% 4822Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 74m 0% 10240Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 60m 0% 10928Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 91m 0% 3052Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 69m 0% 6316Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 160m 0% 11305Mi 10% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 2917Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 62m 0% 9933Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 70m 0% 5457Mi 4% gke-xlou-cdm-ds-0f601751-j488 63m 0% 9444Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2203Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 86m 0% 10234Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 64m 0% 6001Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 91m 0% 4038Mi 6% 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG "5189.633333333333" 03:40:33 DEBUG ] 03:40:33 DEBUG 03:40:33 DEBUG [...] 03:40:33 DEBUG 03:40:33 DEBUG [...] 03:40:33 DEBUG [...] 03:40:33 DEBUG [...] 03:40:33 DEBUG 03:40:33 DEBUG "metric": { 03:40:33 DEBUG [...] 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "value": [ 03:40:33 DEBUG 03:40:33 DEBUG "0" 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:40:33 DEBUG "0.0002666666666667557" 03:40:33 DEBUG --- stderr --- 03:40:33 DEBUG "pod": "ds-idrepo-0" 03:40:33 DEBUG }, 03:40:33 DEBUG } 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "instance": "10.142.0.107:9100" 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG "pod": "ds-idrepo-0" 03:40:33 DEBUG ] 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "0" 03:40:33 DEBUG }, 03:40:33 DEBUG ] 03:40:33 DEBUG 03:40:33 DEBUG "pod": "ds-cts-1" 03:40:33 DEBUG "value": [ 03:40:33 DEBUG ] 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "pod": "ds-idrepo-0" 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG [...] 03:40:33 DEBUG }, 03:40:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "343375872" 03:40:33 DEBUG "0" 03:40:33 DEBUG "0.015" 03:40:33 DEBUG }, 03:40:33 DEBUG } 03:40:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG ] 03:40:33 DEBUG "value": [ 03:40:33 DEBUG } 03:40:33 DEBUG }, 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG } 03:40:33 DEBUG [...] 03:40:33 DEBUG 03:40:33 DEBUG }, 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "value": [ 03:40:33 DEBUG }, 03:40:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:40:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG "value": [ 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG }, 03:40:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:40:33 DEBUG } 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG ] 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "0" 03:40:33 DEBUG } 03:40:33 DEBUG 03:40:33 DEBUG "pod": "ds-cts-1" 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:40:33 DEBUG "instance": "10.142.0.43:9100" 03:40:33 DEBUG }, 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG "value": [ 03:40:33 DEBUG }, 03:40:33 DEBUG }, 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:40:33 DEBUG } 03:40:33 DEBUG "value": [ 03:40:33 DEBUG }, 03:40:33 DEBUG ] 03:40:33 DEBUG "0.000015793583915061034" 03:40:33 DEBUG } 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG ] 03:40:33 DEBUG 03:40:33 DEBUG }, 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG }, 03:40:33 DEBUG "pod": "ds-idrepo-0" 03:40:33 DEBUG "value": [ 03:40:33 DEBUG 03:40:33 DEBUG 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "value": [ 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG "0" 03:40:33 DEBUG }, 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG "value": [ 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG "0" 03:40:33 DEBUG } 03:40:33 DEBUG 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "0" 03:40:33 DEBUG "value": [ 03:40:33 DEBUG }, 03:40:33 INFO [loop_until]: OK (rc = 0) 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:40:33 DEBUG 03:40:33 DEBUG "0" 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG "value": [ 03:40:33 DEBUG "0" 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG ] 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG "value": [ 03:40:33 DEBUG --- stdout --- 03:40:33 DEBUG "2563.5666666666666" 03:40:33 DEBUG }, 03:40:33 DEBUG [...] 03:40:33 DEBUG ] 03:40:33 DEBUG "1" 03:40:33 DEBUG "0" 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG "0" 03:40:33 DEBUG "0" 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG }, 03:40:33 DEBUG "0" 03:40:33 DEBUG } 03:40:33 DEBUG "0" 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG ] 03:40:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6m 8880Mi am-7d597cf7d6-8vr8f 6m 8492Mi am-7d597cf7d6-9wz9h 6m 8711Mi ds-cts-0 13m 7815Mi ds-cts-1 12m 7976Mi ds-cts-2 14m 8107Mi ds-idrepo-0 10m 4577Mi ds-idrepo-1 10m 4288Mi ds-idrepo-2 9m 4103Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 991Mi idm-b85f9d978-rl76f 5m 987Mi lodemon-66fcb5cd8b-xh4df 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 162m 235Mi 03:40:33 DEBUG "value": [ 03:40:33 DEBUG 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG "value": [ 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG "0" 03:40:33 DEBUG } 03:40:33 DEBUG --- stderr --- 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG 03:40:33 DEBUG "0.2313500099679808" 03:40:33 DEBUG 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG "4002.806609178545" 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG "instance": "10.142.0.43:9100" 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG }, 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG "value": [ 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG 1713062433, 03:40:33 DEBUG } 03:40:33 DEBUG } 03:40:33 DEBUG "12667925504" 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG ] 03:40:33 DEBUG } 03:40:33 DEBUG } 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 INFO 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 INFO 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 INFO [loop_until]: kubectl --namespace=xlou top node 03:41:33 INFO [loop_until]: kubectl --namespace=xlou top pods 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713062493" 03:41:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG --- status code --- 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 ERROR [http_cmd]: ERROR 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG --- status code --- 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG --- status code --- 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG --- status code --- 03:41:33 ERROR --- status code --- 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG --- http response --- 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG --- http response --- 03:41:33 ERROR --- http response --- 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:41:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062493" 03:41:33 DEBUG { 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 DEBUG ----- output ----- 03:41:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG { 03:41:33 DEBUG ----- output ----- 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG { 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG { 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG ----- output ----- 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG { 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG { 03:41:33 DEBUG "data": { 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062493" 03:41:33 DEBUG { 03:41:33 DEBUG { 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG { 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG { 03:41:33 DEBUG { 03:41:33 DEBUG { 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 DEBUG { 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062493" 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "data": { 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "data": { 03:41:33 DEBUG "data": { 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "data": { 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "data": { 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG "data": { 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG "resultType": "vector", 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG "data": { 03:41:33 DEBUG "result": [ 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG "data": { 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG "data": { 03:41:33 DEBUG "data": { 03:41:33 DEBUG "data": { 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG { 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG "result": [ 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG { 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG "data": { 03:41:33 DEBUG "result": [ 03:41:33 DEBUG "result": [ 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG "result": [ 03:41:33 DEBUG "resultType": "vector", 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG "resultType": "vector", 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG "result": [ 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "result": [ 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG { 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG "result": [ 03:41:33 DEBUG "metric": { 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG { 03:41:33 DEBUG { 03:41:33 DEBUG "result": [ 03:41:33 DEBUG { 03:41:33 DEBUG "result": [ 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG "result": [ 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG "result": [ 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG { 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:41:33 DEBUG "data": { 03:41:33 DEBUG { 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG "metric": { 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG { 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:41:33 DEBUG "instance": "10.142.0.107:9100" 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG "result": [ 03:41:33 DEBUG { 03:41:33 DEBUG "metric": { 03:41:33 INFO [http_cmd]: http status code OK 03:41:33 DEBUG { 03:41:33 DEBUG "metric": { 03:41:33 DEBUG { 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG { 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG { 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG "metric": { 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG "metric": { 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:41:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:41:33 DEBUG "metric": { 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG }, 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:41:33 DEBUG "metric": { 03:41:33 DEBUG "metric": { 03:41:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:41:33 DEBUG --- status code --- 03:41:33 DEBUG "metric": { 03:41:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:41:33 DEBUG "metric": { 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG "metric": { 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG "metric": { 03:41:33 DEBUG "pod": "ds-idrepo-1" 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:41:33 DEBUG { 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG "result": [ 03:41:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG }, 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG { 03:41:33 DEBUG "value": [ 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG "pod": "ds-idrepo-1" 03:41:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:41:33 DEBUG }, 03:41:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:41:33 DEBUG "instance": "10.142.0.107:9100" 03:41:33 DEBUG }, 03:41:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:41:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:41:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:41:33 DEBUG }, 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG { 03:41:33 DEBUG { 03:41:33 DEBUG }, 03:41:33 DEBUG { 03:41:33 DEBUG "value": [ 03:41:33 DEBUG { 03:41:33 DEBUG }, 03:41:33 DEBUG { 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG 03:41:33 DEBUG { 03:41:33 DEBUG }, 03:41:33 DEBUG }, 03:41:33 DEBUG "value": [ 03:41:33 DEBUG --- http response --- 03:41:33 DEBUG }, 03:41:33 DEBUG "value": [ 03:41:33 DEBUG }, 03:41:33 DEBUG }, 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG }, 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG { 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "data": { 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "metric": { 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "data": { 03:41:33 DEBUG 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 03:41:33 DEBUG "value": [ 03:41:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:41:33 DEBUG ----- output ----- 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "value": [ 03:41:33 DEBUG { 03:41:33 DEBUG "value": [ 03:41:33 DEBUG { 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG "data": { 03:41:33 DEBUG "pod": "ds-idrepo-1" 03:41:33 DEBUG 03:41:33 DEBUG "data": { 03:41:33 DEBUG 03:41:33 DEBUG "data": { 03:41:33 DEBUG 03:41:33 DEBUG "data": { 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG 03:41:33 DEBUG "data": { 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG { 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG "data": { 03:41:33 DEBUG 03:41:33 DEBUG "result": [ 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG "result": [ 03:41:33 DEBUG { 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG "result": [ 03:41:33 DEBUG [...] 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "status": "success", 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "data": { 03:41:33 DEBUG 03:41:33 DEBUG "data": { 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG 03:41:33 DEBUG { 03:41:33 DEBUG }, 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "result": [ 03:41:33 DEBUG [...] 03:41:33 DEBUG "result": [ 03:41:33 DEBUG 03:41:33 DEBUG "result": [ 03:41:33 DEBUG { 03:41:33 DEBUG 03:41:33 DEBUG "result": [ 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG [...] 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "data": { 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG "result": [ 03:41:33 DEBUG [...] 03:41:33 DEBUG "metric": { 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 03:41:33 DEBUG "metric": { 03:41:33 DEBUG { 03:41:33 DEBUG 03:41:33 DEBUG { 03:41:33 DEBUG [...] 03:41:33 DEBUG { 03:41:33 DEBUG "metric": { 03:41:33 DEBUG 03:41:33 DEBUG { 03:41:33 DEBUG [...] 03:41:33 DEBUG [...] 03:41:33 DEBUG 03:41:33 DEBUG [...] 03:41:33 DEBUG 03:41:33 DEBUG "resultType": "vector", 03:41:33 DEBUG [...] 03:41:33 DEBUG [...] 03:41:33 DEBUG "result": [ 03:41:33 DEBUG [...] 03:41:33 DEBUG "result": [ 03:41:33 DEBUG { 03:41:33 DEBUG 03:41:33 DEBUG "instance": "10.142.0.107:9100" 03:41:33 DEBUG [...] 03:41:33 DEBUG 03:41:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:41:33 DEBUG "metric": { 03:41:33 DEBUG 03:41:33 DEBUG "metric": { 03:41:33 DEBUG 03:41:33 DEBUG "metric": { 03:41:33 DEBUG "pod": "ds-idrepo-1" 03:41:33 DEBUG 03:41:33 DEBUG "metric": { 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG [...] 03:41:33 DEBUG "result": [ 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG { 03:41:33 DEBUG 03:41:33 DEBUG { 03:41:33 DEBUG "metric": { 03:41:33 DEBUG 03:41:33 DEBUG }, 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG }, 03:41:33 DEBUG "pod": "ds-idrepo-1" 03:41:33 DEBUG 03:41:33 DEBUG "instance": "10.142.0.107:9100" 03:41:33 DEBUG 03:41:33 DEBUG "pod": "ds-idrepo-1" 03:41:33 DEBUG }, 03:41:33 DEBUG "instance": "10.142.0.43:9100" 03:41:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG { 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "metric": { 03:41:33 DEBUG 03:41:33 DEBUG "metric": { 03:41:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:41:33 DEBUG 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 03:41:33 DEBUG [...] 03:41:33 DEBUG "value": [ 03:41:33 DEBUG }, 03:41:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:41:33 DEBUG }, 03:41:33 DEBUG 03:41:33 DEBUG }, 03:41:33 DEBUG "value": [ 03:41:33 DEBUG }, 03:41:33 DEBUG }, 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "pod": "end-user-ui-58878d7668-zg4h2" 03:41:33 DEBUG "instance": "10.142.0.43:9100" 03:41:33 DEBUG 03:41:33 DEBUG "metric": { 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "pod": "ds-idrepo-1" 03:41:33 DEBUG 03:41:33 DEBUG "instance": "10.142.0.107:9100" 03:41:33 DEBUG }, 03:41:33 DEBUG "pod": "ds-idrepo-0" 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "value": [ 03:41:33 DEBUG }, 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "pod": "ds-cts-1" 03:41:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:41:33 DEBUG }, 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:41:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:41:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:41:33 DEBUG }, 03:41:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:41:33 DEBUG }, 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 03:41:33 DEBUG }, 03:41:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG 03:41:33 DEBUG }, 03:41:33 DEBUG }, 03:41:33 DEBUG 03:41:33 DEBUG }, 03:41:33 DEBUG "0" 03:41:33 DEBUG }, 03:41:33 DEBUG }, 03:41:33 DEBUG }, 03:41:33 DEBUG "value": [ 03:41:33 DEBUG }, 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG }, 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG }, 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "12668554240" 03:41:33 DEBUG 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:41:33 DEBUG "value": [ 03:41:33 DEBUG ] 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG [...] 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 03:41:33 DEBUG [...] 03:41:33 DEBUG "pod": "ds-idrepo-0" 03:41:33 DEBUG [...] 03:41:33 DEBUG 03:41:33 DEBUG "0" 03:41:33 DEBUG "value": [ 03:41:33 DEBUG [...] 03:41:33 DEBUG 03:41:33 DEBUG [...] 03:41:33 DEBUG ] 03:41:33 DEBUG 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG }, 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG } 03:41:33 DEBUG 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG [...] 03:41:33 DEBUG ] 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG "2098.766666666667" 03:41:33 DEBUG [...] 03:41:33 DEBUG 03:41:33 DEBUG } 03:41:33 DEBUG [...] 03:41:33 DEBUG "0" 03:41:33 DEBUG ] 03:41:33 DEBUG "0" 03:41:33 DEBUG "5535.333333333333" 03:41:33 DEBUG ] 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "2079.1" 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG [...] 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "0" 03:41:33 DEBUG }, 03:41:33 DEBUG "850898944" 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG ] 03:41:33 DEBUG 03:41:33 DEBUG "value": [ 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG "33811" 03:41:33 DEBUG [...] 03:41:33 DEBUG ] 03:41:33 DEBUG "3.114197470487693" 03:41:33 DEBUG 03:41:33 DEBUG [...] 03:41:33 DEBUG 03:41:33 DEBUG ] 03:41:33 DEBUG "value": [ 03:41:33 DEBUG ] 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG } 03:41:33 DEBUG 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG 03:41:33 DEBUG "instance": "10.142.0.43:9100" 03:41:33 DEBUG } 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG } 03:41:33 DEBUG "pod": "end-user-ui-58878d7668-zg4h2" 03:41:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:41:33 DEBUG ] 03:41:33 DEBUG "pod": "ds-idrepo-0" 03:41:33 DEBUG 03:41:33 DEBUG 03:41:33 DEBUG "pod": "ds-idrepo-0" 03:41:33 DEBUG } 03:41:33 DEBUG 03:41:33 DEBUG "2079.633333333333" 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG 03:41:33 DEBUG ] 03:41:33 DEBUG 03:41:33 DEBUG ] 03:41:33 DEBUG 03:41:33 DEBUG }, 03:41:33 DEBUG ] 03:41:33 DEBUG "0" 03:41:33 DEBUG ] 03:41:33 DEBUG }, 03:41:33 DEBUG }, 03:41:33 DEBUG } 03:41:33 DEBUG }, 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 03:41:33 DEBUG }, 03:41:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG 03:41:33 DEBUG } 03:41:33 DEBUG 03:41:33 DEBUG } 03:41:33 DEBUG "pod": "ds-idrepo-0" 03:41:33 DEBUG "value": [ 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "instance": "10.142.0.43:9100" 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG "pod": "ds-cts-1" 03:41:33 DEBUG "value": [ 03:41:33 DEBUG }, 03:41:33 DEBUG ] 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:41:33 DEBUG }, 03:41:33 DEBUG 03:41:33 DEBUG } 03:41:33 DEBUG }, 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG ] 03:41:33 DEBUG ] 03:41:33 DEBUG ] 03:41:33 DEBUG "0" 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG }, 03:41:33 DEBUG "0" 03:41:33 DEBUG }, 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG "value": [ 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG "value": [ 03:41:33 DEBUG } 03:41:33 DEBUG "instance": "10.142.0.43:9100" 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "0.00036666666666652263" 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG "value": [ 03:41:33 DEBUG ] 03:41:33 DEBUG "value": [ 03:41:33 DEBUG "0" 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG "0" 03:41:33 DEBUG } 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG }, 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG "0.015" 03:41:33 DEBUG ] 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG } 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG ] 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG "0" 03:41:33 DEBUG "value": [ 03:41:33 DEBUG 1713062493, 03:41:33 DEBUG "0" 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG "0.000029319444444444445" 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG "0" 03:41:33 DEBUG ] 03:41:33 DEBUG "2852.4" 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG ] 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG ] 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 DEBUG } 03:41:33 INFO [loop_until]: OK (rc = 0) 03:41:33 DEBUG --- stdout --- 03:41:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 9751m 7418Mi am-7d597cf7d6-8vr8f 7757m 8554Mi am-7d597cf7d6-9wz9h 11478m 8566Mi ds-cts-0 4077m 7158Mi ds-cts-1 3893m 8002Mi ds-cts-2 4162m 8142Mi ds-idrepo-0 11421m 4581Mi ds-idrepo-1 1842m 4293Mi ds-idrepo-2 1547m 4104Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 991Mi idm-b85f9d978-rl76f 4m 987Mi lodemon-66fcb5cd8b-xh4df 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6320m 1221Mi 03:41:33 DEBUG --- stderr --- 03:41:33 DEBUG 03:41:33 INFO [loop_until]: OK (rc = 0) 03:41:33 DEBUG --- stdout --- 03:41:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 62m 0% 4814Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5129m 17% 10341Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 10418m 34% 9465Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 17872m 59% 3207Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1902m 6% 6361Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 4906m 16% 10659Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2914Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 10172m 34% 9989Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1712m 5% 5509Mi 4% gke-xlou-cdm-ds-0f601751-j488 5064m 16% 9438Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2227Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 13880m 46% 10132Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 12066m 40% 6032Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6643m 41% 5456Mi 9% 03:41:33 DEBUG --- stderr --- 03:41:33 DEBUG 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713062553" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713062553" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062553" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713062553" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG --- http response --- 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062553" 03:42:33 DEBUG ----- output ----- 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG { 03:42:33 DEBUG --- status code --- 03:42:33 ERROR [http_cmd]: ERROR 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 ERROR --- status code --- 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG "data": { 03:42:33 DEBUG --- http response --- 03:42:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG "result": [ 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG --- http response --- 03:42:33 ERROR --- http response --- 03:42:33 DEBUG { 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:42:33 DEBUG "metric": { 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG { 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG --- status code --- 03:42:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG { 03:42:33 DEBUG }, 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG "data": { 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:42:33 DEBUG "value": [ 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG "data": { 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG 03:42:33 DEBUG { 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:42:33 DEBUG { 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:42:33 DEBUG { 03:42:33 DEBUG { 03:42:33 DEBUG { 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG "result": [ 03:42:33 DEBUG { 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG { 03:42:33 DEBUG 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG { 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG "result": [ 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG 03:42:33 DEBUG "data": { 03:42:33 DEBUG { 03:42:33 DEBUG "data": { 03:42:33 DEBUG { 03:42:33 DEBUG "data": { 03:42:33 DEBUG "data": { 03:42:33 DEBUG "data": { 03:42:33 DEBUG "metric": { 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:42:33 DEBUG "data": { 03:42:33 DEBUG { 03:42:33 DEBUG "data": { 03:42:33 DEBUG [...] 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG "pod": "ds-idrepo-1" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG "metric": { 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG 03:42:33 DEBUG "result": [ 03:42:33 DEBUG "data": { 03:42:33 DEBUG "result": [ 03:42:33 DEBUG "data": { 03:42:33 DEBUG "result": [ 03:42:33 DEBUG "result": [ 03:42:33 DEBUG "result": [ 03:42:33 DEBUG }, 03:42:33 DEBUG { 03:42:33 DEBUG "result": [ 03:42:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 DEBUG "result": [ 03:42:33 DEBUG 03:42:33 DEBUG { 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG { 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG { 03:42:33 DEBUG { 03:42:33 DEBUG { 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG { 03:42:33 DEBUG }, 03:42:33 DEBUG { 03:42:33 DEBUG 03:42:33 DEBUG "metric": { 03:42:33 DEBUG "result": [ 03:42:33 DEBUG "metric": { 03:42:33 DEBUG "result": [ 03:42:33 DEBUG "metric": { 03:42:33 DEBUG "metric": { 03:42:33 DEBUG "metric": { 03:42:33 DEBUG 03:42:33 DEBUG "data": { 03:42:33 DEBUG "metric": { 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "metric": { 03:42:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:42:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:42:33 DEBUG { 03:42:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:42:33 DEBUG { 03:42:33 DEBUG "pod": "ds-idrepo-1" 03:42:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:42:33 DEBUG "pod": "ds-idrepo-1" 03:42:33 DEBUG 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG "instance": "10.142.0.107:9100" 03:42:33 DEBUG 03:42:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:42:33 DEBUG }, 03:42:33 DEBUG }, 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG "metric": { 03:42:33 DEBUG }, 03:42:33 DEBUG }, 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 DEBUG "result": [ 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "metric": { 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "value": [ 03:42:33 DEBUG [...] 03:42:33 DEBUG { 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG "instance": "10.142.0.107:9100" 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG "metric": { 03:42:33 DEBUG 03:42:33 DEBUG [...] 03:42:33 DEBUG 03:42:33 DEBUG "8.013090046636945" 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG [...] 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG ] 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG [...] 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG } 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG [...] 03:42:33 DEBUG "pod": "ds-idrepo-0" 03:42:33 DEBUG "value": [ 03:42:33 DEBUG [...] 03:42:33 DEBUG 03:42:33 DEBUG [...] 03:42:33 DEBUG ] 03:42:33 DEBUG 03:42:33 DEBUG [...] 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG [...] 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:42:33 DEBUG 03:42:33 DEBUG } 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:42:33 DEBUG [...] 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG } 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG [...] 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG "pod": "ds-cts-1" 03:42:33 DEBUG 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG "pod": "ds-cts-1" 03:42:33 DEBUG "0" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 DEBUG [...] 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 DEBUG "instance": "10.142.0.43:9100" 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:42:33 DEBUG 03:42:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:42:33 DEBUG "value": [ 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 DEBUG }, 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG --- status code --- 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062553" 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 DEBUG "3.959761244485701" 03:42:33 DEBUG }, 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG "instance": "10.142.0.43:9100" 03:42:33 DEBUG }, 03:42:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:42:33 DEBUG }, 03:42:33 DEBUG }, 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG "value": [ 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062553" 03:42:33 DEBUG ] 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062553" 03:42:33 DEBUG "value": [ 03:42:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062553" 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:42:33 DEBUG "0" 03:42:33 DEBUG }, 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "value": [ 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG "0" 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG 03:42:33 DEBUG ] 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG } 03:42:33 DEBUG 1713062553, 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG ] 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG "0.000033333333333333335" 03:42:33 DEBUG ] 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG ] 03:42:33 DEBUG --- status code --- 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG 03:42:33 DEBUG } 03:42:33 DEBUG "2865.833333333333" 03:42:33 DEBUG ] 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG { 03:42:33 DEBUG { 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG } 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG "0" 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG } 03:42:33 DEBUG "1345.3" 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG } 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:42:33 DEBUG } 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG { 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG ] 03:42:33 DEBUG "12669131776" 03:42:33 DEBUG ] 03:42:33 DEBUG "3097706.891280905" 03:42:33 DEBUG ] 03:42:33 DEBUG ] 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG ] 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG }, 03:42:33 DEBUG "3414831104" 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG "data": { 03:42:33 DEBUG "data": { 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG } 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:42:33 DEBUG } 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG } 03:42:33 DEBUG "data": { 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG ] 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG } 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG "value": [ 03:42:33 DEBUG ] 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG } 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG { 03:42:33 INFO [http_cmd]: http status code OK 03:42:33 DEBUG "result": [ 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG { 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG { 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG } 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:42:33 DEBUG "result": [ 03:42:33 DEBUG { 03:42:33 DEBUG { 03:42:33 DEBUG "result": [ 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG --- status code --- 03:42:33 DEBUG { 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG { 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG "0.015" 03:42:33 DEBUG ] 03:42:33 DEBUG ] 03:42:33 DEBUG { 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG { 03:42:33 DEBUG "data": { 03:42:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:42:33 DEBUG "metric": { 03:42:33 DEBUG { 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG "data": { 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG "data": { 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG "metric": { 03:42:33 DEBUG "data": { 03:42:33 DEBUG "data": { 03:42:33 DEBUG "metric": { 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG --- http response --- 03:42:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG "data": { 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG { 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG "pod": "ds-idrepo-1" 03:42:33 DEBUG "result": [ 03:42:33 DEBUG { 03:42:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:42:33 DEBUG "data": { 03:42:33 DEBUG "result": [ 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG "result": [ 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG } 03:42:33 DEBUG }, 03:42:33 DEBUG "result": [ 03:42:33 DEBUG "result": [ 03:42:33 DEBUG }, 03:42:33 DEBUG }, 03:42:33 DEBUG "metric": { 03:42:33 DEBUG ----- output ----- 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG { 03:42:33 DEBUG "result": [ 03:42:33 DEBUG { 03:42:33 DEBUG "data": { 03:42:33 DEBUG } 03:42:33 DEBUG "value": [ 03:42:33 DEBUG { 03:42:33 DEBUG { 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "instance": "10.142.0.107:9100" 03:42:33 DEBUG { 03:42:33 DEBUG "result": [ 03:42:33 DEBUG "metric": { 03:42:33 DEBUG { 03:42:33 DEBUG "metric": { 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG 03:42:33 DEBUG "metric": { 03:42:33 DEBUG "metric": { 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG "status": "success", 03:42:33 DEBUG { 03:42:33 DEBUG "pod": "ds-idrepo-1" 03:42:33 DEBUG "metric": { 03:42:33 DEBUG "pod": "ds-idrepo-1" 03:42:33 DEBUG "result": [ 03:42:33 DEBUG 03:42:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:42:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "data": { 03:42:33 DEBUG "metric": { 03:42:33 DEBUG }, 03:42:33 DEBUG "pod": "ds-idrepo-1" 03:42:33 DEBUG }, 03:42:33 DEBUG { 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG "resultType": "vector", 03:42:33 DEBUG "instance": "10.142.0.107:9100" 03:42:33 DEBUG "value": [ 03:42:33 DEBUG }, 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "metric": { 03:42:33 DEBUG [...] 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "value": [ 03:42:33 DEBUG [...] 03:42:33 DEBUG [...] 03:42:33 DEBUG 03:42:33 DEBUG "result": [ 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG { 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG [...] 03:42:33 DEBUG "metric": { 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG "instance": "10.142.0.107:9100" 03:42:33 DEBUG 03:42:33 DEBUG [...] 03:42:33 DEBUG 03:42:33 DEBUG [...] 03:42:33 DEBUG 03:42:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:42:33 DEBUG [...] 03:42:33 DEBUG [...] 03:42:33 DEBUG "pod": "ds-idrepo-0" 03:42:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG [...] 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG "value": [ 03:42:33 DEBUG [...] 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "instance": "10.142.0.43:9100" 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG [...] 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG "pod": "ds-idrepo-0" 03:42:33 DEBUG 03:42:33 DEBUG "pod": "ds-idrepo-0" 03:42:33 DEBUG 03:42:33 DEBUG "1345.3" 03:42:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:42:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:42:33 DEBUG "0" 03:42:33 DEBUG "5588.7134482285355" 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG "pod": "ds-idrepo-0" 03:42:33 DEBUG }, 03:42:33 DEBUG 03:42:33 DEBUG ] 03:42:33 DEBUG }, 03:42:33 DEBUG }, 03:42:33 DEBUG ] 03:42:33 DEBUG ] 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG [...] 03:42:33 DEBUG "instance": "10.142.0.43:9100" 03:42:33 DEBUG "value": [ 03:42:33 DEBUG }, 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 03:42:33 DEBUG } 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "value": [ 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG "5197.033333333334" 03:42:33 DEBUG 03:42:33 DEBUG }, 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG "value": [ 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:42:33 DEBUG ] 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG ] 03:42:33 DEBUG ] 03:42:33 DEBUG ] 03:42:33 DEBUG 03:42:33 DEBUG "value": [ 03:42:33 DEBUG "0" 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG "0" 03:42:33 DEBUG }, 03:42:33 DEBUG } 03:42:33 DEBUG "134696" 03:42:33 DEBUG "1345.2" 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG ] 03:42:33 DEBUG "0" 03:42:33 DEBUG ] 03:42:33 DEBUG "value": [ 03:42:33 DEBUG } 03:42:33 DEBUG ] 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG ] 03:42:33 DEBUG "instance": "10.142.0.43:9100" 03:42:33 DEBUG "0.0002666666666667557" 03:42:33 DEBUG ] 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG }, 03:42:33 DEBUG "value": [ 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG ] 03:42:33 DEBUG "0" 03:42:33 DEBUG ] 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG 1713062553, 03:42:33 DEBUG ] 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG ] 03:42:33 DEBUG "0" 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 DEBUG ] 03:42:33 DEBUG } 03:42:33 DEBUG } 03:42:33 INFO 03:42:33 INFO [loop_until]: kubectl --namespace=xlou top pods 03:42:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:42:33 INFO [loop_until]: OK (rc = 0) 03:42:33 DEBUG --- stdout --- 03:42:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5565m 7439Mi am-7d597cf7d6-8vr8f 5272m 8568Mi am-7d597cf7d6-9wz9h 7016m 8589Mi ds-cts-0 2829m 7164Mi ds-cts-1 2911m 7931Mi ds-cts-2 2906m 8144Mi ds-idrepo-0 6928m 4580Mi ds-idrepo-1 1028m 4293Mi ds-idrepo-2 1045m 4104Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 8m 991Mi idm-b85f9d978-rl76f 4m 987Mi lodemon-66fcb5cd8b-xh4df 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8186m 4926Mi 03:42:33 DEBUG --- stderr --- 03:42:33 DEBUG 03:42:33 INFO 03:42:33 INFO [loop_until]: kubectl --namespace=xlou top node 03:42:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:42:33 INFO [loop_until]: OK (rc = 0) 03:42:33 DEBUG --- stdout --- 03:42:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 65m 0% 4808Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2692m 9% 10335Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 5960m 19% 9487Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 11473m 38% 3595Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1163m 3% 6354Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 2822m 9% 10634Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2914Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 5755m 19% 10007Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1099m 3% 5508Mi 4% gke-xlou-cdm-ds-0f601751-j488 2977m 9% 9434Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 61m 0% 2220Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 7695m 25% 10128Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 6828m 22% 6055Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8154m 51% 8321Mi 14% 03:42:33 DEBUG --- stderr --- 03:42:33 DEBUG 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713062613" 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062613" 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713062613" 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG --- status code --- 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 ERROR [http_cmd]: ERROR 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 DEBUG --- status code --- 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 DEBUG --- status code --- 03:43:33 ERROR --- status code --- 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:43:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG --- http response --- 03:43:33 ERROR --- http response --- 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG { 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG --- status code --- 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG { 03:43:33 DEBUG "data": { 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG "data": { 03:43:33 DEBUG "result": [ 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG { 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG { 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG "result": [ 03:43:33 DEBUG "metric": { 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:43:33 DEBUG { 03:43:33 DEBUG { 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:43:33 DEBUG { 03:43:33 DEBUG { 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG { 03:43:33 DEBUG { 03:43:33 DEBUG "instance": "10.142.0.107:9100" 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "data": { 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "metric": { 03:43:33 DEBUG }, 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG { 03:43:33 DEBUG "data": { 03:43:33 DEBUG "data": { 03:43:33 DEBUG { 03:43:33 DEBUG "data": { 03:43:33 DEBUG "data": { 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG "data": { 03:43:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:43:33 DEBUG "value": [ 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG "result": [ 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG }, 03:43:33 DEBUG 03:43:33 DEBUG "data": { 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:43:33 DEBUG "result": [ 03:43:33 DEBUG "result": [ 03:43:33 DEBUG "data": { 03:43:33 DEBUG "result": [ 03:43:33 DEBUG "result": [ 03:43:33 DEBUG { 03:43:33 DEBUG "result": [ 03:43:33 DEBUG "value": [ 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG { 03:43:33 DEBUG { 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG { 03:43:33 DEBUG { 03:43:33 DEBUG "metric": { 03:43:33 DEBUG { 03:43:33 DEBUG 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG 03:43:33 DEBUG "result": [ 03:43:33 DEBUG { 03:43:33 DEBUG "metric": { 03:43:33 DEBUG "metric": { 03:43:33 DEBUG "result": [ 03:43:33 DEBUG "metric": { 03:43:33 DEBUG "metric": { 03:43:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:43:33 DEBUG "metric": { 03:43:33 DEBUG 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG [...] 03:43:33 DEBUG { 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:43:33 DEBUG "pod": "ds-idrepo-1" 03:43:33 DEBUG { 03:43:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:43:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:43:33 DEBUG }, 03:43:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:43:33 DEBUG 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG 03:43:33 DEBUG "metric": { 03:43:33 DEBUG "pod": "ds-idrepo-1" 03:43:33 DEBUG "data": { 03:43:33 DEBUG }, 03:43:33 DEBUG "metric": { 03:43:33 DEBUG }, 03:43:33 DEBUG }, 03:43:33 DEBUG "value": [ 03:43:33 DEBUG }, 03:43:33 DEBUG [...] 03:43:33 DEBUG 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:43:33 DEBUG }, 03:43:33 DEBUG }, 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "pod": "ds-idrepo-1" 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "result": [ 03:43:33 DEBUG 03:43:33 DEBUG }, 03:43:33 DEBUG 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062613" 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG "instance": "10.142.0.43:9100" 03:43:33 DEBUG { 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG { 03:43:33 DEBUG 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG }, 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG "metric": { 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG [...] 03:43:33 DEBUG 03:43:33 DEBUG [...] 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "data": { 03:43:33 DEBUG 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 DEBUG 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG "pod": "ds-idrepo-1" 03:43:33 DEBUG [...] 03:43:33 DEBUG 03:43:33 DEBUG [...] 03:43:33 DEBUG [...] 03:43:33 DEBUG 03:43:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:43:33 DEBUG 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG "resultType": "vector", 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062613" 03:43:33 DEBUG [...] 03:43:33 DEBUG [...] 03:43:33 DEBUG --- status code --- 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 DEBUG }, 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713062613" 03:43:33 DEBUG 03:43:33 DEBUG }, 03:43:33 DEBUG 03:43:33 DEBUG "2788.266666666667" 03:43:33 DEBUG "result": [ 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 03:43:33 DEBUG [...] 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 03:43:33 DEBUG ] 03:43:33 DEBUG { 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:43:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:43:33 DEBUG } 03:43:33 DEBUG "metric": { 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:43:33 DEBUG "pod": "ds-cts-1" 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG }, 03:43:33 DEBUG }, 03:43:33 DEBUG "8.034196784364816" 03:43:33 DEBUG }, 03:43:33 DEBUG ] 03:43:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:43:33 DEBUG }, 03:43:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:43:33 DEBUG }, 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG }, 03:43:33 DEBUG 03:43:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "value": [ 03:43:33 DEBUG ] 03:43:33 DEBUG "value": [ 03:43:33 DEBUG } 03:43:33 DEBUG "pod": "ds-idrepo-0" 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 03:43:33 DEBUG "value": [ 03:43:33 DEBUG { 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "pod": "ds-idrepo-0" 03:43:33 DEBUG }, 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG 1713062613, 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG } 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG } 03:43:33 DEBUG }, 03:43:33 DEBUG 03:43:33 DEBUG [...] 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG }, 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "1311.7333333333333" 03:43:33 DEBUG "0.015" 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG ] 03:43:33 DEBUG "0" 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 DEBUG "value": [ 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG 03:43:33 DEBUG "0" 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG "data": { 03:43:33 DEBUG "0.000029319444444444445" 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG ] 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062613" 03:43:33 DEBUG ] 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG } 03:43:33 DEBUG ] 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG --- status code --- 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062613" 03:43:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062613" 03:43:33 DEBUG 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG 03:43:33 DEBUG ] 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG ] 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG "6.770079544814225" 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG "0" 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG [...] 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG 03:43:33 DEBUG } 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG "result": [ 03:43:33 DEBUG } 03:43:33 DEBUG "0" 03:43:33 DEBUG ] 03:43:33 DEBUG ] 03:43:33 DEBUG ] 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:43:33 DEBUG ] 03:43:33 DEBUG ] 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG "pod": "ds-cts-1" 03:43:33 DEBUG ] 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG { 03:43:33 DEBUG ] 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:43:33 DEBUG }, 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:43:33 DEBUG } 03:43:33 DEBUG "metric": { 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:43:33 DEBUG } 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG } 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG { 03:43:33 INFO [http_cmd]: http status code OK 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG { 03:43:33 DEBUG } 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG "value": [ 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG } 03:43:33 DEBUG "pod": "ds-idrepo-1" 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG } 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG --- status code --- 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG 03:43:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:43:33 DEBUG { 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG { 03:43:33 DEBUG }, 03:43:33 DEBUG { 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG "data": { 03:43:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:43:33 DEBUG } 03:43:33 DEBUG }, 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "0" 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:43:33 DEBUG } 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:43:33 DEBUG --- http response --- 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG "data": { 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "data": { 03:43:33 DEBUG ] 03:43:33 DEBUG "data": { 03:43:33 DEBUG 03:43:33 DEBUG "data": { 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:43:33 DEBUG "result": [ 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG { 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG } 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG { 03:43:33 DEBUG { 03:43:33 DEBUG { 03:43:33 DEBUG { 03:43:33 DEBUG ----- output ----- 03:43:33 DEBUG { 03:43:33 DEBUG "result": [ 03:43:33 DEBUG { 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "253755518.44966477" 03:43:33 DEBUG "result": [ 03:43:33 DEBUG ] 03:43:33 DEBUG "result": [ 03:43:33 DEBUG { 03:43:33 DEBUG "result": [ 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG { 03:43:33 DEBUG "metric": { 03:43:33 DEBUG { 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "data": { 03:43:33 DEBUG ] 03:43:33 DEBUG { 03:43:33 DEBUG } 03:43:33 DEBUG "metric": { 03:43:33 DEBUG 03:43:33 DEBUG { 03:43:33 DEBUG "data": { 03:43:33 DEBUG "data": { 03:43:33 DEBUG "data": { 03:43:33 DEBUG "data": { 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG "status": "success", 03:43:33 DEBUG "metric": { 03:43:33 DEBUG "data": { 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG } 03:43:33 DEBUG "metric": { 03:43:33 DEBUG } 03:43:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:43:33 DEBUG [...] 03:43:33 DEBUG "metric": { 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:43:33 DEBUG "result": [ 03:43:33 DEBUG "data": { 03:43:33 DEBUG "instance": "10.142.0.107:9100" 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG "result": [ 03:43:33 DEBUG ] 03:43:33 DEBUG "instance": "10.142.0.107:9100" 03:43:33 DEBUG }, 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:43:33 DEBUG "result": [ 03:43:33 DEBUG "result": [ 03:43:33 DEBUG "result": [ 03:43:33 DEBUG }, 03:43:33 DEBUG { 03:43:33 DEBUG "resultType": "vector", 03:43:33 DEBUG }, 03:43:33 DEBUG "result": [ 03:43:33 DEBUG { 03:43:33 DEBUG } 03:43:33 DEBUG }, 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG }, 03:43:33 DEBUG { 03:43:33 DEBUG { 03:43:33 DEBUG { 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "metric": { 03:43:33 DEBUG "result": [ 03:43:33 DEBUG "value": [ 03:43:33 DEBUG { 03:43:33 DEBUG "metric": { 03:43:33 DEBUG } 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "metric": { 03:43:33 DEBUG "metric": { 03:43:33 DEBUG 03:43:33 DEBUG "instance": "10.142.0.107:9100" 03:43:33 DEBUG { 03:43:33 DEBUG 03:43:33 DEBUG "metric": { 03:43:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG "metric": { 03:43:33 DEBUG "pod": "ds-idrepo-0" 03:43:33 DEBUG 03:43:33 DEBUG "pod": "ds-idrepo-1" 03:43:33 DEBUG "pod": "ds-idrepo-1" 03:43:33 DEBUG }, 03:43:33 DEBUG }, 03:43:33 DEBUG "metric": { 03:43:33 DEBUG 03:43:33 DEBUG "instance": "10.142.0.107:9100" 03:43:33 DEBUG }, 03:43:33 DEBUG 03:43:33 DEBUG [...] 03:43:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:43:33 DEBUG }, 03:43:33 DEBUG 03:43:33 DEBUG }, 03:43:33 DEBUG 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:43:33 DEBUG 03:43:33 DEBUG }, 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG }, 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG }, 03:43:33 DEBUG [...] 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 03:43:33 DEBUG [...] 03:43:33 DEBUG 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG [...] 03:43:33 DEBUG 03:43:33 DEBUG [...] 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG "0" 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:43:33 DEBUG 03:43:33 DEBUG ] 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG [...] 03:43:33 DEBUG [...] 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG [...] 03:43:33 DEBUG 03:43:33 DEBUG }, 03:43:33 DEBUG 03:43:33 DEBUG } 03:43:33 DEBUG 03:43:33 DEBUG [...] 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG [...] 03:43:33 DEBUG [...] 03:43:33 DEBUG 03:43:33 DEBUG "instance": "10.142.0.43:9100" 03:43:33 DEBUG "value": [ 03:43:33 DEBUG [...] 03:43:33 DEBUG ] 03:43:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:43:33 DEBUG 03:43:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG "instance": "10.142.0.43:9100" 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG }, 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG 03:43:33 DEBUG } 03:43:33 DEBUG }, 03:43:33 DEBUG 03:43:33 DEBUG }, 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG }, 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "3127437.1021350906" 03:43:33 DEBUG 03:43:33 DEBUG } 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "pod": "ds-idrepo-0" 03:43:33 DEBUG "instance": "10.142.0.43:9100" 03:43:33 DEBUG "value": [ 03:43:33 DEBUG 03:43:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:43:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG ] 03:43:33 DEBUG 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG "pod": "ds-idrepo-0" 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG }, 03:43:33 DEBUG }, 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG 03:43:33 DEBUG }, 03:43:33 DEBUG }, 03:43:33 DEBUG "12669746176" 03:43:33 DEBUG } 03:43:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:43:33 DEBUG "6762975232" 03:43:33 DEBUG }, 03:43:33 DEBUG "1311.9" 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "0" 03:43:33 DEBUG "instance": "10.142.0.43:9100" 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "value": [ 03:43:33 DEBUG ] 03:43:33 DEBUG ] 03:43:33 DEBUG }, 03:43:33 DEBUG ] 03:43:33 DEBUG "value": [ 03:43:33 DEBUG ] 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG ] 03:43:33 DEBUG }, 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG "value": [ 03:43:33 DEBUG } 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG } 03:43:33 DEBUG "0" 03:43:33 DEBUG "5427.833333333334" 03:43:33 DEBUG } 03:43:33 DEBUG "value": [ 03:43:33 DEBUG "213430" 03:43:33 DEBUG "1311.8666666666666" 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG "0" 03:43:33 DEBUG ] 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG 1713062613, 03:43:33 DEBUG ] 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG "0" 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG ] 03:43:33 DEBUG ] 03:43:33 DEBUG "0.00033333333333350387" 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG ] 03:43:33 DEBUG ] 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG ] 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG ] 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 DEBUG } 03:43:33 INFO 03:43:33 INFO [loop_until]: kubectl --namespace=xlou top pods 03:43:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:43:33 INFO [loop_until]: OK (rc = 0) 03:43:33 DEBUG --- stdout --- 03:43:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5555m 7441Mi am-7d597cf7d6-8vr8f 5180m 8572Mi am-7d597cf7d6-9wz9h 6740m 8591Mi ds-cts-0 2693m 7190Mi ds-cts-1 2768m 7937Mi ds-cts-2 2807m 8158Mi ds-idrepo-0 6469m 4580Mi ds-idrepo-1 1068m 4293Mi ds-idrepo-2 1006m 4104Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 991Mi idm-b85f9d978-rl76f 4m 987Mi lodemon-66fcb5cd8b-xh4df 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8045m 7277Mi 03:43:33 DEBUG --- stderr --- 03:43:33 DEBUG 03:43:33 INFO 03:43:33 INFO [loop_until]: kubectl --namespace=xlou top node 03:43:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:43:33 INFO [loop_until]: OK (rc = 0) 03:43:33 DEBUG --- stdout --- 03:43:33 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4816Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 2807m 9% 10355Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 5784m 19% 9484Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 11432m 38% 3600Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1079m 3% 6342Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 2758m 9% 10637Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 64m 0% 2907Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 5677m 19% 10011Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1088m 3% 5510Mi 4% gke-xlou-cdm-ds-0f601751-j488 2880m 9% 9421Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2228Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 7486m 25% 10139Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 6619m 22% 6059Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 7917m 49% 10344Mi 17% 03:43:33 DEBUG --- stderr --- 03:43:33 DEBUG 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 ERROR [http_cmd]: ERROR 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG --- status code --- 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 ERROR --- status code --- 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG --- status code --- 03:44:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 ERROR --- http response --- 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:44:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG { 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG { 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG { 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:44:33 DEBUG "data": { 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:44:33 DEBUG "data": { 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:44:33 DEBUG "data": { 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG { 03:44:33 DEBUG "result": [ 03:44:33 DEBUG { 03:44:33 DEBUG "result": [ 03:44:33 DEBUG { 03:44:33 DEBUG "result": [ 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG { 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG { 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG { 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 DEBUG "data": { 03:44:33 DEBUG "metric": { 03:44:33 DEBUG "data": { 03:44:33 DEBUG "metric": { 03:44:33 DEBUG "data": { 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG "resultType": "vector", 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:44:33 DEBUG "metric": { 03:44:33 DEBUG "result": [ 03:44:33 DEBUG "result": [ 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG }, 03:44:33 DEBUG "result": [ 03:44:33 DEBUG }, 03:44:33 DEBUG "instance": "10.142.0.107:9100" 03:44:33 DEBUG { 03:44:33 DEBUG { 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG "value": [ 03:44:33 DEBUG { 03:44:33 DEBUG "value": [ 03:44:33 DEBUG }, 03:44:33 DEBUG "metric": { 03:44:33 DEBUG "metric": { 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG 03:44:33 DEBUG "metric": { 03:44:33 DEBUG 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:44:33 DEBUG "pod": "ds-idrepo-1" 03:44:33 DEBUG 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:44:33 DEBUG "pod": "ds-idrepo-1" 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG }, 03:44:33 DEBUG }, 03:44:33 DEBUG 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG }, 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "value": [ 03:44:33 DEBUG [...] 03:44:33 DEBUG { 03:44:33 DEBUG "value": [ 03:44:33 DEBUG [...] 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG [...] 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "data": { 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG [...] 03:44:33 DEBUG [...] 03:44:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:44:33 DEBUG "result": [ 03:44:33 DEBUG [...] 03:44:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG }, 03:44:33 DEBUG { 03:44:33 DEBUG "metric": { 03:44:33 DEBUG }, 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 DEBUG "instance": "10.142.0.43:9100" 03:44:33 DEBUG 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 03:44:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:44:33 DEBUG "value": [ 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG }, 03:44:33 DEBUG 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG 03:44:33 DEBUG }, 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG "pod": "ds-idrepo-0" 03:44:33 DEBUG "2076.133333333333" 03:44:33 DEBUG 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "1066.0333333333333" 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG }, 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG }, 03:44:33 DEBUG ] 03:44:33 DEBUG "pod": "ds-idrepo-0" 03:44:33 DEBUG 03:44:33 DEBUG ] 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG "2994.4333333333334" 03:44:33 DEBUG "value": [ 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 03:44:33 DEBUG "value": [ 03:44:33 DEBUG } 03:44:33 DEBUG }, 03:44:33 DEBUG 03:44:33 DEBUG } 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:44:33 DEBUG ] 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG ] 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 03:44:33 DEBUG ] 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG } 03:44:33 DEBUG "5.12757227907687" 03:44:33 DEBUG { 03:44:33 DEBUG { 03:44:33 DEBUG "0" 03:44:33 DEBUG } 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG [...] 03:44:33 DEBUG } 03:44:33 DEBUG { 03:44:33 DEBUG ] 03:44:33 DEBUG ] 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG "0" 03:44:33 DEBUG 03:44:33 DEBUG } 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG "data": { 03:44:33 DEBUG "data": { 03:44:33 DEBUG } 03:44:33 DEBUG ] 03:44:33 DEBUG 03:44:33 DEBUG "data": { 03:44:33 DEBUG } 03:44:33 DEBUG ] 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG "resultType": "vector", 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG ] 03:44:33 DEBUG 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG } 03:44:33 DEBUG "result": [ 03:44:33 DEBUG "result": [ 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG } 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG } 03:44:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:44:33 DEBUG "result": [ 03:44:33 DEBUG } 03:44:33 DEBUG { 03:44:33 DEBUG { 03:44:33 DEBUG --- status code --- 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG ] 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG } 03:44:33 DEBUG }, 03:44:33 DEBUG { 03:44:33 DEBUG "metric": { 03:44:33 DEBUG "metric": { 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG } 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "metric": { 03:44:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:44:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG } 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG "pod": "ds-idrepo-1" 03:44:33 DEBUG }, 03:44:33 DEBUG }, 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:44:33 DEBUG { 03:44:33 DEBUG "238143.38242110543" 03:44:33 DEBUG }, 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "value": [ 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG ] 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG { 03:44:33 DEBUG "data": { 03:44:33 DEBUG } 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG { 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG ] 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG "data": { 03:44:33 DEBUG "result": [ 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 DEBUG } 03:44:33 DEBUG 03:44:33 DEBUG [...] 03:44:33 DEBUG [...] 03:44:33 DEBUG "data": { 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG { 03:44:33 DEBUG } 03:44:33 DEBUG [...] 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG "result": [ 03:44:33 DEBUG "metric": { 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "result": [ 03:44:33 DEBUG { 03:44:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG { 03:44:33 DEBUG "metric": { 03:44:33 DEBUG }, 03:44:33 DEBUG 03:44:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:44:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:44:33 DEBUG "metric": { 03:44:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "pod": "ds-cts-1" 03:44:33 DEBUG }, 03:44:33 DEBUG }, 03:44:33 DEBUG "pod": "ds-idrepo-1" 03:44:33 DEBUG }, 03:44:33 DEBUG 03:44:33 DEBUG }, 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "value": [ 03:44:33 DEBUG }, 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG "0" 03:44:33 DEBUG "0" 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG [...] 03:44:33 DEBUG "0.00017591666666666667" 03:44:33 DEBUG ] 03:44:33 DEBUG ] 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG 03:44:33 DEBUG [...] 03:44:33 DEBUG 03:44:33 DEBUG } 03:44:33 DEBUG ] 03:44:33 DEBUG ] 03:44:33 DEBUG [...] 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG }, 03:44:33 DEBUG } 03:44:33 DEBUG 03:44:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "pod": "ds-idrepo-0" 03:44:33 DEBUG }, 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG }, 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "254047" 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG ] 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG "8.017194288272114" 03:44:33 DEBUG } 03:44:33 DEBUG "0" 03:44:33 DEBUG ] 03:44:33 DEBUG ] 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG { 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG "data": { 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG "result": [ 03:44:33 DEBUG { 03:44:33 DEBUG "metric": { 03:44:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:44:33 DEBUG }, 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG [...] 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:44:33 DEBUG }, 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG "0.015" 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713062673" 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062673" 03:44:33 DEBUG --- status code --- 03:44:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062673" 03:44:33 DEBUG --- status code --- 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG --- status code --- 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG --- status code --- 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG { 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:44:33 DEBUG { 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG { 03:44:33 DEBUG "data": { 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:44:33 DEBUG { 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG "data": { 03:44:33 DEBUG { 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG "data": { 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG "result": [ 03:44:33 DEBUG { 03:44:33 DEBUG "data": { 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG "result": [ 03:44:33 DEBUG "data": { 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:44:33 DEBUG --- status code --- 03:44:33 INFO [http_cmd]: http status code OK 03:44:33 DEBUG { 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG { 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG "result": [ 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG --- status code --- 03:44:33 DEBUG "metric": { 03:44:33 DEBUG "data": { 03:44:33 DEBUG "result": [ 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:44:33 DEBUG "metric": { 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:44:33 DEBUG "result": [ 03:44:33 DEBUG { 03:44:33 DEBUG { 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:44:33 DEBUG "instance": "10.142.0.107:9100" 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG { 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG "pod": "ds-idrepo-1" 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG { 03:44:33 DEBUG "metric": { 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG --- http response --- 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:44:33 DEBUG }, 03:44:33 DEBUG "result": [ 03:44:33 DEBUG "metric": { 03:44:33 DEBUG { 03:44:33 DEBUG }, 03:44:33 DEBUG { 03:44:33 DEBUG "metric": { 03:44:33 DEBUG "pod": "ds-idrepo-1" 03:44:33 DEBUG "data": { 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:44:33 DEBUG "value": [ 03:44:33 DEBUG { 03:44:33 DEBUG "instance": "10.142.0.107:9100" 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:44:33 DEBUG }, 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG { 03:44:33 DEBUG ----- output ----- 03:44:33 DEBUG 03:44:33 DEBUG "metric": { 03:44:33 DEBUG }, 03:44:33 DEBUG "data": { 03:44:33 DEBUG 03:44:33 DEBUG "data": { 03:44:33 DEBUG }, 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "result": [ 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG { 03:44:33 DEBUG 03:44:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 03:44:33 DEBUG { 03:44:33 DEBUG "data": { 03:44:33 DEBUG "status": "success", 03:44:33 DEBUG 03:44:33 DEBUG }, 03:44:33 DEBUG 03:44:33 DEBUG "result": [ 03:44:33 DEBUG 03:44:33 DEBUG "result": [ 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "metric": { 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG "data": { 03:44:33 DEBUG [...] 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 03:44:33 DEBUG { 03:44:33 DEBUG [...] 03:44:33 DEBUG { 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:44:33 DEBUG "result": [ 03:44:33 DEBUG "resultType": "vector", 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "metric": { 03:44:33 DEBUG 03:44:33 DEBUG "metric": { 03:44:33 DEBUG 03:44:33 DEBUG [...] 03:44:33 DEBUG }, 03:44:33 DEBUG { 03:44:33 DEBUG "result": [ 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG [...] 03:44:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:44:33 DEBUG 03:44:33 DEBUG "instance": "10.142.0.107:9100" 03:44:33 DEBUG [...] 03:44:33 DEBUG 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "metric": { 03:44:33 DEBUG { 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG }, 03:44:33 DEBUG 03:44:33 DEBUG }, 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "pod": "ds-idrepo-1" 03:44:33 DEBUG "metric": { 03:44:33 DEBUG "instance": "10.142.0.43:9100" 03:44:33 DEBUG [...] 03:44:33 DEBUG 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "pod": "ds-cts-1" 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG }, 03:44:33 DEBUG "instance": "10.142.0.107:9100" 03:44:33 DEBUG }, 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG }, 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "pod": "ds-idrepo-0" 03:44:33 DEBUG 03:44:33 DEBUG "value": [ 03:44:33 DEBUG }, 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 03:44:33 DEBUG "instance": "10.142.0.43:9100" 03:44:33 DEBUG 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 03:44:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:44:33 DEBUG }, 03:44:33 DEBUG [...] 03:44:33 DEBUG 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG 03:44:33 DEBUG }, 03:44:33 DEBUG 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG 03:44:33 DEBUG }, 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG "12670383104" 03:44:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:44:33 DEBUG "value": [ 03:44:33 DEBUG [...] 03:44:33 DEBUG "0" 03:44:33 DEBUG [...] 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG ] 03:44:33 DEBUG }, 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG 03:44:33 DEBUG ] 03:44:33 DEBUG 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG "0" 03:44:33 DEBUG 03:44:33 DEBUG [...] 03:44:33 DEBUG 03:44:33 DEBUG } 03:44:33 DEBUG "value": [ 03:44:33 DEBUG "0.00020000000000000757" 03:44:33 DEBUG 03:44:33 DEBUG } 03:44:33 DEBUG 03:44:33 DEBUG "1874621.127310549" 03:44:33 DEBUG ] 03:44:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:44:33 DEBUG 03:44:33 DEBUG [...] 03:44:33 DEBUG ] 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG ] 03:44:33 DEBUG 03:44:33 DEBUG ] 03:44:33 DEBUG 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG }, 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG } 03:44:33 DEBUG "1065.7666666666667" 03:44:33 DEBUG } 03:44:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:44:33 DEBUG } 03:44:33 DEBUG "instance": "10.142.0.43:9100" 03:44:33 DEBUG } 03:44:33 DEBUG ] 03:44:33 DEBUG "value": [ 03:44:33 DEBUG 03:44:33 DEBUG 03:44:33 DEBUG } 03:44:33 DEBUG ] 03:44:33 DEBUG ] 03:44:33 DEBUG }, 03:44:33 DEBUG "value": [ 03:44:33 DEBUG }, 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG "pod": "ds-idrepo-0" 03:44:33 DEBUG 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG "value": [ 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG "7984619520" 03:44:33 DEBUG ] 03:44:33 DEBUG }, 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG "1065.7666666666667" 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG "0" 03:44:33 DEBUG "instance": "10.142.0.43:9100" 03:44:33 DEBUG } 03:44:33 DEBUG "value": [ 03:44:33 DEBUG } 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG ] 03:44:33 DEBUG }, 03:44:33 DEBUG ] 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG "value": [ 03:44:33 DEBUG } 03:44:33 DEBUG "0" 03:44:33 DEBUG ] 03:44:33 DEBUG ] 03:44:33 DEBUG 1713062673, 03:44:33 DEBUG } 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG "5561.866666666667" 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG ] 03:44:33 DEBUG ] 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 DEBUG } 03:44:33 INFO 03:44:33 INFO [loop_until]: kubectl --namespace=xlou top pods 03:44:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:44:33 INFO [loop_until]: OK (rc = 0) 03:44:33 DEBUG --- stdout --- 03:44:33 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6700m 7458Mi am-7d597cf7d6-8vr8f 6249m 8579Mi am-7d597cf7d6-9wz9h 8945m 8717Mi ds-cts-0 7109m 7193Mi ds-cts-1 7230m 7943Mi ds-cts-2 7010m 8162Mi ds-idrepo-0 6316m 4580Mi ds-idrepo-1 937m 4293Mi ds-idrepo-2 1046m 4104Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 990Mi idm-b85f9d978-rl76f 3m 987Mi lodemon-66fcb5cd8b-xh4df 2m 62Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7052m 16Mi 03:44:33 DEBUG --- stderr --- 03:44:33 DEBUG 03:44:33 INFO 03:44:33 INFO [loop_until]: kubectl --namespace=xlou top node 03:44:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:44:34 INFO [loop_until]: OK (rc = 0) 03:44:34 DEBUG --- stdout --- 03:44:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 61m 0% 4816Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 7296m 24% 10327Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6453m 21% 9519Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 8608m 28% 3112Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 980m 3% 6342Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 7213m 24% 10611Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 65m 0% 2901Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 6033m 20% 10038Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 984m 3% 5521Mi 4% gke-xlou-cdm-ds-0f601751-j488 7547m 25% 9436Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2216Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 7619m 25% 10588Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 5486m 18% 6055Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 6440m 40% 3822Mi 6% 03:44:34 DEBUG --- stderr --- 03:44:34 DEBUG 03:44:37 WARNING File /tmp/lodemon_data-20240414_033733/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [14/Apr/2024 03:44:37] "GET /monitoring/average?start_time=20240414_024221&stop_time=20240414_024236 HTTP/1.1" 200 - 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713062733" 03:45:33 ERROR [http_cmd]: ERROR 03:45:33 ERROR --- status code --- 03:45:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:45:33 ERROR --- http response --- 03:45:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:45:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG --- http response --- 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG --- http response --- 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062733" 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG { 03:45:33 DEBUG { 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG "data": { 03:45:33 DEBUG "data": { 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG "result": [ 03:45:33 DEBUG "result": [ 03:45:33 DEBUG { 03:45:33 DEBUG { 03:45:33 DEBUG "metric": { 03:45:33 DEBUG "metric": { 03:45:33 DEBUG "pod": "ds-idrepo-1" 03:45:33 DEBUG "pod": "ds-idrepo-1" 03:45:33 DEBUG }, 03:45:33 DEBUG }, 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG [...] 03:45:33 DEBUG [...] 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "pod": "ds-idrepo-0" 03:45:33 DEBUG "pod": "ds-cts-1" 03:45:33 DEBUG }, 03:45:33 DEBUG }, 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG "0" 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG "0.00006666666666666667" 03:45:33 DEBUG ] 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG } 03:45:33 DEBUG ] 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG { 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG "data": { 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG "result": [ 03:45:33 DEBUG { 03:45:33 DEBUG "metric": { 03:45:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:45:33 DEBUG }, 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG [...] 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:45:33 DEBUG }, 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG "1.630861122912554" 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG { 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG "data": { 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG "result": [ 03:45:33 DEBUG { 03:45:33 DEBUG "metric": { 03:45:33 DEBUG "instance": "10.142.0.107:9100" 03:45:33 DEBUG }, 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG [...] 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "instance": "10.142.0.43:9100" 03:45:33 DEBUG }, 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG "3110.733333333333" 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG { 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG "data": { 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:45:33 DEBUG "result": [ 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG { 03:45:33 DEBUG { 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG "metric": { 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG "data": { 03:45:33 DEBUG "resultType": "vector", 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG }, 03:45:33 DEBUG --- status code --- 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 DEBUG "result": [ 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG "value": [ 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG { 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG "metric": { 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG --- status code --- 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG 03:45:33 DEBUG { 03:45:33 DEBUG }, 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG [...] 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG "value": [ 03:45:33 DEBUG { 03:45:33 DEBUG { 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:45:33 DEBUG 03:45:33 DEBUG "data": { 03:45:33 DEBUG 03:45:33 DEBUG "status": "success", 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:45:33 DEBUG 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG 03:45:33 DEBUG "data": { 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG "data": { 03:45:33 DEBUG { 03:45:33 DEBUG { 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG 03:45:33 DEBUG "result": [ 03:45:33 DEBUG 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG { 03:45:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:45:33 DEBUG { 03:45:33 DEBUG [...] 03:45:33 DEBUG "result": [ 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG "result": [ 03:45:33 DEBUG "data": { 03:45:33 DEBUG "data": { 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG "data": { 03:45:33 DEBUG "metric": { 03:45:33 DEBUG 03:45:33 DEBUG { 03:45:33 DEBUG { 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG }, 03:45:33 DEBUG "resultType": "vector", 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG "pod": "ds-idrepo-1" 03:45:33 DEBUG 03:45:33 DEBUG "metric": { 03:45:33 DEBUG "metric": { 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG "result": [ 03:45:33 DEBUG "result": [ 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "result": [ 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG }, 03:45:33 DEBUG 03:45:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:45:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:45:33 DEBUG { 03:45:33 DEBUG { 03:45:33 DEBUG { 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG { 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:45:33 DEBUG }, 03:45:33 DEBUG }, 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG "metric": { 03:45:33 DEBUG "metric": { 03:45:33 DEBUG "861.1" 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG "metric": { 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG 03:45:33 DEBUG }, 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "data": { 03:45:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:45:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:45:33 DEBUG ] 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:45:33 DEBUG }, 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG }, 03:45:33 DEBUG }, 03:45:33 DEBUG } 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG 03:45:33 DEBUG "value": [ 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "result": [ 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "value": [ 03:45:33 DEBUG ] 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG { 03:45:33 DEBUG "1342.2666666666667" 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG { 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG } 03:45:33 DEBUG [...] 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:45:33 DEBUG 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG ] 03:45:33 DEBUG [...] 03:45:33 DEBUG [...] 03:45:33 DEBUG "metric": { 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG 03:45:33 DEBUG "data": { 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG { 03:45:33 DEBUG [...] 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG ] 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG }, 03:45:33 DEBUG [...] 03:45:33 DEBUG [...] 03:45:33 DEBUG 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG 03:45:33 DEBUG "result": [ 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "pod": "ds-idrepo-0" 03:45:33 DEBUG "data": { 03:45:33 DEBUG 03:45:33 DEBUG { 03:45:33 DEBUG } 03:45:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:45:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG }, 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG 03:45:33 DEBUG "metric": { 03:45:33 DEBUG }, 03:45:33 DEBUG }, 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "result": [ 03:45:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:45:33 DEBUG "pod": "ds-idrepo-1" 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 03:45:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:45:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG { 03:45:33 DEBUG }, 03:45:33 DEBUG }, 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG [...] 03:45:33 DEBUG }, 03:45:33 DEBUG }, 03:45:33 DEBUG "0" 03:45:33 DEBUG "metric": { 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "0" 03:45:33 DEBUG "0.9551071311075219" 03:45:33 DEBUG 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "value": [ 03:45:33 DEBUG ] 03:45:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG 03:45:33 DEBUG ] 03:45:33 DEBUG ] 03:45:33 DEBUG 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG } 03:45:33 DEBUG }, 03:45:33 DEBUG "64410.39165193275" 03:45:33 DEBUG 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG "0" 03:45:33 DEBUG "869.0333333333333" 03:45:33 DEBUG ] 03:45:33 DEBUG "value": [ 03:45:33 DEBUG ] 03:45:33 DEBUG 03:45:33 DEBUG ] 03:45:33 DEBUG ] 03:45:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:45:33 DEBUG ] 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG } 03:45:33 DEBUG [...] 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 DEBUG }, 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG ] 03:45:33 DEBUG 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 DEBUG "value": [ 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG } 03:45:33 DEBUG ] 03:45:33 DEBUG [...] 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG "182054.5" 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG "pod": "ds-idrepo-0" 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG }, 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG "value": [ 03:45:33 DEBUG ] 03:45:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG } 03:45:33 DEBUG }, 03:45:33 DEBUG "0" 03:45:33 DEBUG } 03:45:33 DEBUG "value": [ 03:45:33 DEBUG ] 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG } 03:45:33 DEBUG "0.015" 03:45:33 DEBUG ] 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG { 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG "data": { 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG "result": [ 03:45:33 DEBUG { 03:45:33 DEBUG "metric": { 03:45:33 DEBUG "pod": "ds-idrepo-1" 03:45:33 DEBUG }, 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG [...] 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "pod": "ds-idrepo-0" 03:45:33 DEBUG }, 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG "0" 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG { 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG "data": { 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG "result": [ 03:45:33 DEBUG { 03:45:33 DEBUG "metric": { 03:45:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:45:33 DEBUG }, 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG [...] 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:45:33 DEBUG }, 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG "1648297.4241636333" 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062733" 03:45:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062733" 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG { 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG { 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG { 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG "data": { 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG "data": { 03:45:33 DEBUG "resultType": "vector", 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG "data": { 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:45:33 DEBUG "result": [ 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG "result": [ 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG { 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG "result": [ 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG { 03:45:33 DEBUG { 03:45:33 DEBUG { 03:45:33 DEBUG "metric": { 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG { 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG "metric": { 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG "instance": "10.142.0.107:9100" 03:45:33 DEBUG "metric": { 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:45:33 DEBUG "data": { 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:45:33 DEBUG "pod": "ds-idrepo-1" 03:45:33 INFO [http_cmd]: http status code OK 03:45:33 DEBUG "data": { 03:45:33 DEBUG }, 03:45:33 DEBUG "instance": "10.142.0.107:9100" 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG }, 03:45:33 DEBUG --- status code --- 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG "value": [ 03:45:33 DEBUG }, 03:45:33 DEBUG { 03:45:33 DEBUG "result": [ 03:45:33 DEBUG { 03:45:33 DEBUG "value": [ 03:45:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:45:33 DEBUG "result": [ 03:45:33 DEBUG 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG { 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG 03:45:33 DEBUG --- http response --- 03:45:33 DEBUG { 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "data": { 03:45:33 DEBUG "data": { 03:45:33 DEBUG 03:45:33 DEBUG "metric": { 03:45:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:45:33 DEBUG 03:45:33 DEBUG "metric": { 03:45:33 DEBUG 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG 03:45:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:45:33 DEBUG ----- output ----- 03:45:33 DEBUG [...] 03:45:33 DEBUG "pod": "ds-idrepo-1" 03:45:33 DEBUG [...] 03:45:33 DEBUG "result": [ 03:45:33 DEBUG "result": [ 03:45:33 DEBUG [...] 03:45:33 DEBUG }, 03:45:33 DEBUG { 03:45:33 DEBUG 03:45:33 DEBUG }, 03:45:33 DEBUG 03:45:33 DEBUG { 03:45:33 DEBUG { 03:45:33 DEBUG 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "status": "success", 03:45:33 DEBUG 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 03:45:33 DEBUG "metric": { 03:45:33 DEBUG "metric": { 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "data": { 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:45:33 DEBUG "instance": "10.142.0.107:9100" 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "resultType": "vector", 03:45:33 DEBUG "instance": "10.142.0.43:9100" 03:45:33 DEBUG 03:45:33 DEBUG "instance": "10.142.0.43:9100" 03:45:33 DEBUG }, 03:45:33 DEBUG }, 03:45:33 DEBUG "pod": "ds-cts-1" 03:45:33 DEBUG 03:45:33 DEBUG "result": [ 03:45:33 DEBUG }, 03:45:33 DEBUG 03:45:33 DEBUG }, 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "value": [ 03:45:33 DEBUG }, 03:45:33 DEBUG [...] 03:45:33 DEBUG { 03:45:33 DEBUG "value": [ 03:45:33 DEBUG [...] 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 03:45:33 DEBUG "metric": { 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG 03:45:33 DEBUG "instance": "10.142.0.107:9100" 03:45:33 DEBUG "0.00020000000000000757" 03:45:33 DEBUG 03:45:33 DEBUG "12670987264" 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG "0" 03:45:33 DEBUG 03:45:33 DEBUG }, 03:45:33 DEBUG ] 03:45:33 DEBUG 03:45:33 DEBUG ] 03:45:33 DEBUG [...] 03:45:33 DEBUG [...] 03:45:33 DEBUG ] 03:45:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:45:33 DEBUG "value": [ 03:45:33 DEBUG } 03:45:33 DEBUG "pod": "ds-idrepo-0" 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG } 03:45:33 DEBUG }, 03:45:33 DEBUG 03:45:33 DEBUG ] 03:45:33 DEBUG }, 03:45:33 DEBUG ] 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG ] 03:45:33 DEBUG "value": [ 03:45:33 DEBUG 03:45:33 DEBUG } 03:45:33 DEBUG "value": [ 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG 03:45:33 DEBUG } 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG 03:45:33 DEBUG } 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG } 03:45:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:45:33 DEBUG "instance": "10.142.0.43:9100" 03:45:33 DEBUG } 03:45:33 DEBUG "861.3666666666667" 03:45:33 DEBUG [...] 03:45:33 DEBUG "0" 03:45:33 DEBUG }, 03:45:33 DEBUG }, 03:45:33 DEBUG ] 03:45:33 DEBUG 03:45:33 DEBUG ] 03:45:33 DEBUG "value": [ 03:45:33 DEBUG "value": [ 03:45:33 DEBUG } 03:45:33 DEBUG 03:45:33 DEBUG } 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG ] 03:45:33 DEBUG 03:45:33 DEBUG ] 03:45:33 DEBUG "545513472" 03:45:33 DEBUG "0" 03:45:33 DEBUG } 03:45:33 DEBUG "instance": "10.142.0.43:9100" 03:45:33 DEBUG } 03:45:33 DEBUG ] 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG }, 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 DEBUG "value": [ 03:45:33 DEBUG ] 03:45:33 DEBUG ] 03:45:33 DEBUG 1713062733, 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 DEBUG "5575.733333333334" 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG ] 03:45:33 DEBUG } 03:45:33 DEBUG } 03:45:33 INFO 03:45:33 INFO [loop_until]: kubectl --namespace=xlou top pods 03:45:33 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:45:34 INFO 03:45:34 INFO [loop_until]: kubectl --namespace=xlou top node 03:45:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:45:34 INFO [loop_until]: OK (rc = 0) 03:45:34 DEBUG --- stdout --- 03:45:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 5859m 7459Mi am-7d597cf7d6-8vr8f 3177m 8583Mi am-7d597cf7d6-9wz9h 4215m 9114Mi ds-cts-0 4220m 7195Mi ds-cts-1 5690m 7944Mi ds-cts-2 6128m 8130Mi ds-idrepo-0 5235m 4580Mi ds-idrepo-1 820m 4292Mi ds-idrepo-2 683m 4104Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 990Mi idm-b85f9d978-rl76f 4m 987Mi lodemon-66fcb5cd8b-xh4df 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 3684m 477Mi 03:45:34 DEBUG --- stderr --- 03:45:34 DEBUG 03:45:34 INFO [loop_until]: OK (rc = 0) 03:45:34 DEBUG --- stdout --- 03:45:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 66m 0% 4823Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 4703m 15% 10319Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6821m 22% 9510Mi 8% gke-xlou-cdm-default-pool-1b0769c8-qkns 7610m 25% 3050Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 1050m 3% 6361Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 6480m 21% 10602Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 66m 0% 2915Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 6902m 23% 10030Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 854m 2% 5509Mi 4% gke-xlou-cdm-ds-0f601751-j488 6469m 21% 9427Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 64m 0% 2222Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 10597m 35% 10677Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 7890m 26% 6060Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 4705m 29% 4335Mi 7% 03:45:34 DEBUG --- stderr --- 03:45:34 DEBUG 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 ERROR [http_cmd]: ERROR 03:46:33 ERROR --- status code --- 03:46:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:46:33 ERROR --- http response --- 03:46:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:46:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG --- http response --- 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062793" 03:46:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG --- status code --- 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG --- http response --- 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG { 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:46:33 DEBUG { 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:46:33 DEBUG { 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG "data": { 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG { 03:46:33 DEBUG "data": { 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG { 03:46:33 DEBUG { 03:46:33 DEBUG "data": { 03:46:33 DEBUG { 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG "result": [ 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG "data": { 03:46:33 DEBUG "result": [ 03:46:33 DEBUG { 03:46:33 DEBUG "data": { 03:46:33 DEBUG "data": { 03:46:33 DEBUG "result": [ 03:46:33 DEBUG "data": { 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG { 03:46:33 DEBUG "metric": { 03:46:33 DEBUG "resultType": "vector", 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG { 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG "result": [ 03:46:33 DEBUG "metric": { 03:46:33 DEBUG "pod": "ds-idrepo-1" 03:46:33 DEBUG "result": [ 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG "result": [ 03:46:33 DEBUG "metric": { 03:46:33 DEBUG "result": [ 03:46:33 DEBUG { 03:46:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:46:33 DEBUG }, 03:46:33 DEBUG { 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG { 03:46:33 DEBUG "pod": "ds-idrepo-1" 03:46:33 DEBUG { 03:46:33 DEBUG "metric": { 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG "metric": { 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG "metric": { 03:46:33 DEBUG }, 03:46:33 DEBUG "metric": { 03:46:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:46:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:46:33 DEBUG "value": [ 03:46:33 DEBUG "pod": "ds-cts-2" 03:46:33 DEBUG }, 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG }, 03:46:33 DEBUG }, 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "value": [ 03:46:33 DEBUG "value": [ 03:46:33 DEBUG { 03:46:33 DEBUG 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG [...] 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG [...] 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "data": { 03:46:33 DEBUG [...] 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG [...] 03:46:33 DEBUG 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG 03:46:33 DEBUG [...] 03:46:33 DEBUG [...] 03:46:33 DEBUG "result": [ 03:46:33 DEBUG 03:46:33 DEBUG [...] 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG "pod": "ds-cts-1" 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG { 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG }, 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "metric": { 03:46:33 DEBUG "pod": "ds-idrepo-0" 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "instance": "10.142.0.107:9100" 03:46:33 DEBUG }, 03:46:33 DEBUG 03:46:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:46:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:46:33 DEBUG }, 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG "0.00012129166666666667" 03:46:33 DEBUG { 03:46:33 DEBUG }, 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG "0.015" 03:46:33 DEBUG ] 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG "value": [ 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG "0" 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG "data": { 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG 03:46:33 DEBUG ] 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG "191215" 03:46:33 DEBUG } 03:46:33 DEBUG ] 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG "0" 03:46:33 DEBUG "7.915371993540905" 03:46:33 DEBUG 03:46:33 DEBUG } 03:46:33 DEBUG "176701.05373223245" 03:46:33 DEBUG ] 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG "result": [ 03:46:33 DEBUG ] 03:46:33 DEBUG ] 03:46:33 DEBUG [...] 03:46:33 DEBUG ] 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 DEBUG { 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 DEBUG 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG "metric": { 03:46:33 DEBUG ] 03:46:33 DEBUG ] 03:46:33 DEBUG 03:46:33 DEBUG } 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG "pod": "ds-idrepo-1" 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 DEBUG 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 DEBUG }, 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 DEBUG "instance": "10.142.0.43:9100" 03:46:33 DEBUG } 03:46:33 DEBUG "value": [ 03:46:33 DEBUG }, 03:46:33 DEBUG 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG 03:46:33 DEBUG "3034.266666666667" 03:46:33 DEBUG [...] 03:46:33 DEBUG ] 03:46:33 DEBUG 03:46:33 DEBUG } 03:46:33 DEBUG 03:46:33 DEBUG ] 03:46:33 DEBUG 03:46:33 DEBUG } 03:46:33 DEBUG "pod": "ds-idrepo-0" 03:46:33 DEBUG } 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG "0" 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG ----- output ----- 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG { 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG { 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG { 03:46:33 DEBUG { 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG "status": "success", 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG "data": { 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG "data": { 03:46:33 DEBUG "data": { 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG "data": { 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG "result": [ 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG "result": [ 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG "result": [ 03:46:33 DEBUG { 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG "result": [ 03:46:33 DEBUG { 03:46:33 DEBUG { 03:46:33 DEBUG { 03:46:33 DEBUG { 03:46:33 DEBUG { 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG { 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:46:33 DEBUG "metric": { 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG "metric": { 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG "metric": { 03:46:33 DEBUG "data": { 03:46:33 DEBUG "metric": { 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG "instance": "10.142.0.107:9100" 03:46:33 DEBUG "data": { 03:46:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:46:33 DEBUG "data": { 03:46:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG "pod": "ds-idrepo-1" 03:46:33 DEBUG { 03:46:33 DEBUG }, 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG }, 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG }, 03:46:33 DEBUG "result": [ 03:46:33 DEBUG }, 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG "value": [ 03:46:33 DEBUG "result": [ 03:46:33 DEBUG "value": [ 03:46:33 DEBUG "result": [ 03:46:33 DEBUG "value": [ 03:46:33 DEBUG { 03:46:33 DEBUG "value": [ 03:46:33 DEBUG "data": { 03:46:33 DEBUG 03:46:33 DEBUG { 03:46:33 DEBUG 03:46:33 DEBUG { 03:46:33 DEBUG 03:46:33 DEBUG "metric": { 03:46:33 DEBUG 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG 03:46:33 DEBUG "metric": { 03:46:33 DEBUG 03:46:33 DEBUG "metric": { 03:46:33 DEBUG 03:46:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:46:33 DEBUG 03:46:33 DEBUG "result": [ 03:46:33 DEBUG 03:46:33 DEBUG "pod": "ds-cts-2" 03:46:33 DEBUG 03:46:33 DEBUG "pod": "ds-cts-2" 03:46:33 DEBUG 03:46:33 DEBUG }, 03:46:33 DEBUG 03:46:33 DEBUG { 03:46:33 DEBUG [...] 03:46:33 DEBUG }, 03:46:33 DEBUG [...] 03:46:33 DEBUG }, 03:46:33 DEBUG [...] 03:46:33 DEBUG "value": [ 03:46:33 DEBUG [...] 03:46:33 DEBUG "metric": { 03:46:33 DEBUG 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "pod": "ds-idrepo-1" 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG }, 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "value": [ 03:46:33 DEBUG "instance": "10.142.0.43:9100" 03:46:33 DEBUG 03:46:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:46:33 DEBUG 03:46:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:46:33 DEBUG [...] 03:46:33 DEBUG "pod": "ds-idrepo-0" 03:46:33 DEBUG 03:46:33 DEBUG }, 03:46:33 DEBUG [...] 03:46:33 DEBUG }, 03:46:33 DEBUG [...] 03:46:33 DEBUG }, 03:46:33 DEBUG 03:46:33 DEBUG }, 03:46:33 DEBUG 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG [...] 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 DEBUG "12671509504" 03:46:33 DEBUG 03:46:33 DEBUG "1316.0333333333333" 03:46:33 DEBUG 03:46:33 DEBUG "1312.9333333333334" 03:46:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:46:33 DEBUG "0" 03:46:33 DEBUG 03:46:33 DEBUG ] 03:46:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:46:33 DEBUG ] 03:46:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:46:33 DEBUG ] 03:46:33 DEBUG }, 03:46:33 DEBUG ] 03:46:33 DEBUG 03:46:33 DEBUG } 03:46:33 DEBUG }, 03:46:33 DEBUG } 03:46:33 DEBUG }, 03:46:33 DEBUG } 03:46:33 DEBUG "value": [ 03:46:33 DEBUG } 03:46:33 DEBUG 03:46:33 DEBUG ] 03:46:33 DEBUG "value": [ 03:46:33 DEBUG ] 03:46:33 DEBUG "value": [ 03:46:33 DEBUG ] 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG ] 03:46:33 DEBUG "pod": "ds-idrepo-0" 03:46:33 DEBUG } 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG } 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG } 03:46:33 DEBUG "1311.8" 03:46:33 DEBUG } 03:46:33 DEBUG }, 03:46:33 DEBUG } 03:46:33 DEBUG "6.1882889430397645" 03:46:33 DEBUG } 03:46:33 DEBUG "0" 03:46:33 DEBUG } 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG "value": [ 03:46:33 DEBUG ] 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 DEBUG ] 03:46:33 DEBUG "0" 03:46:33 DEBUG ] 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG { 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG "data": { 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG "result": [ 03:46:33 DEBUG { 03:46:33 DEBUG "metric": { 03:46:33 DEBUG "pod": "ds-cts-2" 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG [...] 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG "0" 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG { 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG "data": { 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG "result": [ 03:46:33 DEBUG { 03:46:33 DEBUG "metric": { 03:46:33 DEBUG "pod": "ds-idrepo-1" 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG [...] 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "pod": "ds-cts-1" 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG "0" 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713062793" 03:46:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062793" 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG { 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG "data": { 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG "result": [ 03:46:33 DEBUG { 03:46:33 DEBUG "metric": { 03:46:33 DEBUG "pod": "ds-idrepo-1" 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG [...] 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "pod": "ds-idrepo-0" 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG "0" 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG { 03:46:33 DEBUG { 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG "data": { 03:46:33 DEBUG "data": { 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG "result": [ 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG "result": [ 03:46:33 DEBUG { 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG { 03:46:33 DEBUG "metric": { 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG "metric": { 03:46:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG "instance": "10.142.0.107:9100" 03:46:33 DEBUG }, 03:46:33 DEBUG }, 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:46:33 DEBUG "value": [ 03:46:33 DEBUG "value": [ 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG { 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "data": { 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG [...] 03:46:33 DEBUG [...] 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "result": [ 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG { 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "metric": { 03:46:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:46:33 DEBUG "instance": "10.142.0.43:9100" 03:46:33 DEBUG "instance": "10.142.0.107:9100" 03:46:33 DEBUG }, 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG }, 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG { 03:46:33 DEBUG "value": [ 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG 03:46:33 DEBUG "1315.3" 03:46:33 DEBUG "data": { 03:46:33 DEBUG "0.00006666666666674815" 03:46:33 DEBUG 03:46:33 DEBUG ] 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG ] 03:46:33 DEBUG 03:46:33 DEBUG } 03:46:33 DEBUG "result": [ 03:46:33 DEBUG } 03:46:33 DEBUG [...] 03:46:33 DEBUG ] 03:46:33 DEBUG { 03:46:33 DEBUG ] 03:46:33 DEBUG 03:46:33 DEBUG } 03:46:33 DEBUG "metric": { 03:46:33 DEBUG } 03:46:33 DEBUG 03:46:33 DEBUG } 03:46:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:46:33 DEBUG } 03:46:33 DEBUG 03:46:33 DEBUG }, 03:46:33 DEBUG "instance": "10.142.0.43:9100" 03:46:33 DEBUG "value": [ 03:46:33 DEBUG }, 03:46:33 DEBUG 03:46:33 INFO [http_cmd]: http status code OK 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 03:46:33 DEBUG --- status code --- 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG 03:46:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:46:33 DEBUG "5670.233333333334" 03:46:33 DEBUG [...] 03:46:33 DEBUG --- http response --- 03:46:33 DEBUG ] 03:46:33 DEBUG 03:46:33 DEBUG } 03:46:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:46:33 DEBUG 03:46:33 DEBUG ] 03:46:33 DEBUG ----- output ----- 03:46:33 DEBUG 03:46:33 DEBUG } 03:46:33 DEBUG { 03:46:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:46:33 DEBUG } 03:46:33 DEBUG "status": "success", 03:46:33 DEBUG }, 03:46:33 DEBUG "data": { 03:46:33 DEBUG "value": [ 03:46:33 DEBUG "resultType": "vector", 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG "result": [ 03:46:33 DEBUG "3095629824" 03:46:33 DEBUG { 03:46:33 DEBUG ] 03:46:33 DEBUG "metric": { 03:46:33 DEBUG } 03:46:33 DEBUG "instance": "10.142.0.107:9100" 03:46:33 DEBUG ] 03:46:33 DEBUG }, 03:46:33 DEBUG } 03:46:33 DEBUG "value": [ 03:46:33 DEBUG } 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG [...] 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG 03:46:33 DEBUG "instance": "10.142.0.43:9100" 03:46:33 DEBUG }, 03:46:33 DEBUG "value": [ 03:46:33 DEBUG 1713062793, 03:46:33 DEBUG "0" 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG ] 03:46:33 DEBUG } 03:46:33 DEBUG } 03:46:34 INFO 03:46:34 INFO [loop_until]: kubectl --namespace=xlou top pods 03:46:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:46:34 INFO 03:46:34 INFO [loop_until]: kubectl --namespace=xlou top node 03:46:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:46:34 INFO [loop_until]: OK (rc = 0) 03:46:34 DEBUG --- stdout --- 03:46:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 7427m 7461Mi am-7d597cf7d6-8vr8f 6186m 8580Mi am-7d597cf7d6-9wz9h 8479m 9123Mi ds-cts-0 5838m 7197Mi ds-cts-1 6182m 7920Mi ds-cts-2 5935m 8163Mi ds-idrepo-0 7354m 4581Mi ds-idrepo-1 1134m 4293Mi ds-idrepo-2 1091m 4104Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 5m 990Mi idm-b85f9d978-rl76f 4m 987Mi lodemon-66fcb5cd8b-xh4df 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 8027m 4284Mi 03:46:34 DEBUG --- stderr --- 03:46:34 DEBUG 03:46:34 INFO [loop_until]: OK (rc = 0) 03:46:34 DEBUG --- stdout --- 03:46:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4822Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5985m 20% 10319Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 7645m 25% 10988Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 11326m 37% 3625Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1200m 4% 6345Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 5955m 19% 10629Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2917Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 6918m 23% 10025Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1170m 3% 5514Mi 4% gke-xlou-cdm-ds-0f601751-j488 6162m 20% 9436Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2220Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 9124m 30% 10681Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 7351m 24% 6058Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 7899m 49% 7983Mi 13% 03:46:34 DEBUG --- stderr --- 03:46:34 DEBUG 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 ERROR [http_cmd]: ERROR 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 ERROR --- status code --- 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG --- status code --- 03:47:33 ERROR --- http response --- 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG --- http response --- 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:47:33 DEBUG --- status code --- 03:47:33 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:47:33 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG { 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG { 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG { 03:47:33 DEBUG { 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG { 03:47:33 DEBUG { 03:47:33 DEBUG { 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG "data": { 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG "data": { 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "data": { 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:47:33 DEBUG "data": { 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:47:33 DEBUG "data": { 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:47:33 DEBUG "data": { 03:47:33 DEBUG "data": { 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG { 03:47:33 DEBUG { 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG "result": [ 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "result": [ 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG { 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG "result": [ 03:47:33 DEBUG { 03:47:33 DEBUG "result": [ 03:47:33 DEBUG { 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:47:33 DEBUG "result": [ 03:47:33 DEBUG { 03:47:33 DEBUG "result": [ 03:47:33 DEBUG "result": [ 03:47:33 DEBUG { 03:47:33 DEBUG "data": { 03:47:33 DEBUG "data": { 03:47:33 DEBUG { 03:47:33 DEBUG "data": { 03:47:33 DEBUG { 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG "data": { 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG { 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG { 03:47:33 DEBUG { 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "metric": { 03:47:33 DEBUG { 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "data": { 03:47:33 DEBUG { 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "data": { 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "pod": "ds-cts-2" 03:47:33 DEBUG "result": [ 03:47:33 DEBUG "result": [ 03:47:33 DEBUG "instance": "10.142.0.107:9100" 03:47:33 DEBUG "result": [ 03:47:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "result": [ 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:47:33 DEBUG "pod": "ds-idrepo-1" 03:47:33 DEBUG }, 03:47:33 DEBUG { 03:47:33 DEBUG { 03:47:33 DEBUG }, 03:47:33 DEBUG { 03:47:33 DEBUG }, 03:47:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:47:33 DEBUG { 03:47:33 DEBUG "result": [ 03:47:33 DEBUG "data": { 03:47:33 DEBUG }, 03:47:33 DEBUG "result": [ 03:47:33 DEBUG }, 03:47:33 DEBUG }, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "value": [ 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "value": [ 03:47:33 DEBUG }, 03:47:33 DEBUG "metric": { 03:47:33 DEBUG { 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "value": [ 03:47:33 DEBUG { 03:47:33 DEBUG "value": [ 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 03:47:33 DEBUG "pod": "ds-idrepo-1" 03:47:33 DEBUG "instance": "10.142.0.107:9100" 03:47:33 DEBUG 03:47:33 DEBUG "pod": "ds-cts-2" 03:47:33 DEBUG 03:47:33 DEBUG "value": [ 03:47:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "result": [ 03:47:33 DEBUG 03:47:33 DEBUG "metric": { 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG }, 03:47:33 DEBUG }, 03:47:33 DEBUG 03:47:33 DEBUG }, 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG }, 03:47:33 DEBUG "pod": "ds-idrepo-1" 03:47:33 DEBUG { 03:47:33 DEBUG 03:47:33 DEBUG "pod": "ds-idrepo-1" 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG "value": [ 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG "value": [ 03:47:33 DEBUG }, 03:47:33 DEBUG "metric": { 03:47:33 DEBUG 03:47:33 DEBUG }, 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG "value": [ 03:47:33 DEBUG "pod": "ds-idrepo-1" 03:47:33 DEBUG [...] 03:47:33 DEBUG "value": [ 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG }, 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG "instance": "10.142.0.43:9100" 03:47:33 DEBUG 03:47:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:47:33 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG }, 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG }, 03:47:33 DEBUG 03:47:33 DEBUG }, 03:47:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG }, 03:47:33 DEBUG }, 03:47:33 DEBUG 03:47:33 DEBUG "pod": "ds-idrepo-0" 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG "value": [ 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 03:47:33 DEBUG }, 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG "pod": "ds-idrepo-0" 03:47:33 DEBUG "instance": "10.142.0.43:9100" 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG "pod": "lodemon-66fcb5cd8b-xh4df" 03:47:33 DEBUG }, 03:47:33 DEBUG "192019.5" 03:47:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG "pod": "ds-idrepo-0" 03:47:33 DEBUG "value": [ 03:47:33 DEBUG "0" 03:47:33 DEBUG }, 03:47:33 DEBUG }, 03:47:33 DEBUG "2945.5" 03:47:33 DEBUG }, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG ] 03:47:33 DEBUG }, 03:47:33 DEBUG "pod": "ds-cts-1" 03:47:33 DEBUG 03:47:33 DEBUG "0" 03:47:33 DEBUG "8.025901314844315" 03:47:33 DEBUG }, 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG ] 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG ] 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG } 03:47:33 DEBUG "value": [ 03:47:33 DEBUG }, 03:47:33 DEBUG 03:47:33 DEBUG ] 03:47:33 DEBUG ] 03:47:33 DEBUG "value": [ 03:47:33 DEBUG "0" 03:47:33 DEBUG } 03:47:33 DEBUG "value": [ 03:47:33 DEBUG "0" 03:47:33 DEBUG } 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG "1244.7333333333333" 03:47:33 DEBUG ] 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG "pod": "ds-idrepo-0" 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG ] 03:47:33 DEBUG ] 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG "2051.8327936323044" 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG "0.015" 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG }, 03:47:33 DEBUG ] 03:47:33 DEBUG ] 03:47:33 DEBUG "0" 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG "12672101376" 03:47:33 DEBUG ] 03:47:33 DEBUG ] 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG "0.00006666666666666667" 03:47:33 DEBUG "value": [ 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG "0" 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG { 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG "data": { 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "result": [ 03:47:33 DEBUG { 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "pod": "ds-idrepo-1" 03:47:33 DEBUG }, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG "pod": "ds-idrepo-0" 03:47:33 DEBUG }, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG "0" 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (152 lines): 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG { 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG "data": { 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "result": [ 03:47:33 DEBUG { 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "pod": "ds-cts-2" 03:47:33 DEBUG }, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:47:33 DEBUG }, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG "0" 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG { 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG "data": { 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "result": [ 03:47:33 DEBUG { 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:47:33 DEBUG }, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:47:33 DEBUG }, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG "1069.8" 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713062853" 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG { 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG "data": { 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "result": [ 03:47:33 DEBUG { 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "pod": "ds-cts-2" 03:47:33 DEBUG }, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG "pod": "lodemon-66fcb5cd8b-xh4df" 03:47:33 DEBUG }, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG "0" 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG { 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG "data": { 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "result": [ 03:47:33 DEBUG { 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:47:33 DEBUG }, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:47:33 DEBUG }, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG "1244.7" 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:33 INFO [http_cmd]: http status code OK 03:47:33 DEBUG --- status code --- 03:47:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:33 DEBUG --- http response --- 03:47:33 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:47:33 DEBUG ----- output ----- 03:47:33 DEBUG { 03:47:33 DEBUG "status": "success", 03:47:33 DEBUG "data": { 03:47:33 DEBUG "resultType": "vector", 03:47:33 DEBUG "result": [ 03:47:33 DEBUG { 03:47:33 DEBUG "metric": { 03:47:33 DEBUG "pod": "ds-idrepo-1" 03:47:33 DEBUG }, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG [...] 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG 03:47:33 DEBUG "pod": "ds-cts-1" 03:47:33 DEBUG }, 03:47:33 DEBUG "value": [ 03:47:33 DEBUG 1713062853, 03:47:33 DEBUG "0" 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG ] 03:47:33 DEBUG } 03:47:33 DEBUG } 03:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062854" 03:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062854" 03:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062854" 03:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713062854" 03:47:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062854" 03:47:34 INFO [http_cmd]: http status code OK 03:47:34 INFO [http_cmd]: http status code OK 03:47:34 DEBUG --- status code --- 03:47:34 INFO [http_cmd]: http status code OK 03:47:34 DEBUG --- status code --- 03:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:34 DEBUG --- status code --- 03:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:34 DEBUG --- http response --- 03:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:34 DEBUG --- http response --- 03:47:34 DEBUG --- http response --- 03:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:47:34 DEBUG ----- output ----- 03:47:34 DEBUG ----- output ----- 03:47:34 DEBUG ----- output ----- 03:47:34 DEBUG { 03:47:34 DEBUG { 03:47:34 DEBUG { 03:47:34 DEBUG "status": "success", 03:47:34 DEBUG "status": "success", 03:47:34 DEBUG "status": "success", 03:47:34 DEBUG "data": { 03:47:34 DEBUG "data": { 03:47:34 DEBUG "data": { 03:47:34 DEBUG "resultType": "vector", 03:47:34 DEBUG "resultType": "vector", 03:47:34 DEBUG "resultType": "vector", 03:47:34 DEBUG "result": [ 03:47:34 DEBUG "result": [ 03:47:34 DEBUG "result": [ 03:47:34 DEBUG { 03:47:34 DEBUG { 03:47:34 DEBUG { 03:47:34 DEBUG "metric": { 03:47:34 DEBUG "metric": { 03:47:34 DEBUG "metric": { 03:47:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:47:34 DEBUG "instance": "10.142.0.107:9100" 03:47:34 DEBUG "instance": "10.142.0.107:9100" 03:47:34 DEBUG }, 03:47:34 DEBUG }, 03:47:34 DEBUG }, 03:47:34 DEBUG "value": [ 03:47:34 DEBUG "value": [ 03:47:34 DEBUG "value": [ 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG [...] 03:47:34 DEBUG [...] 03:47:34 DEBUG [...] 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:47:34 INFO [http_cmd]: http status code OK 03:47:34 DEBUG "instance": "10.142.0.43:9100" 03:47:34 DEBUG "instance": "10.142.0.43:9100" 03:47:34 DEBUG }, 03:47:34 DEBUG --- status code --- 03:47:34 DEBUG }, 03:47:34 INFO [http_cmd]: http status code OK 03:47:34 DEBUG }, 03:47:34 DEBUG "value": [ 03:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:34 DEBUG "value": [ 03:47:34 DEBUG --- status code --- 03:47:34 DEBUG "value": [ 03:47:34 DEBUG 1713062854, 03:47:34 DEBUG --- http response --- 03:47:34 DEBUG 1713062854, 03:47:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:47:34 DEBUG 1713062854, 03:47:34 DEBUG "1244.7" 03:47:34 DEBUG "5617.033333333333" 03:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:47:34 DEBUG --- http response --- 03:47:34 DEBUG "0.000333333333333267" 03:47:34 DEBUG ] 03:47:34 DEBUG ] 03:47:34 DEBUG ----- output ----- 03:47:34 DEBUG ] 03:47:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:47:34 DEBUG } 03:47:34 DEBUG } 03:47:34 DEBUG { 03:47:34 DEBUG } 03:47:34 DEBUG ----- output ----- 03:47:34 DEBUG ] 03:47:34 DEBUG ] 03:47:34 DEBUG "status": "success", 03:47:34 DEBUG ] 03:47:34 DEBUG { 03:47:34 DEBUG } 03:47:34 DEBUG } 03:47:34 DEBUG "data": { 03:47:34 DEBUG } 03:47:34 DEBUG "status": "success", 03:47:34 DEBUG } 03:47:34 DEBUG } 03:47:34 DEBUG "resultType": "vector", 03:47:34 DEBUG } 03:47:34 DEBUG "data": { 03:47:34 DEBUG "result": [ 03:47:34 DEBUG "resultType": "vector", 03:47:34 DEBUG { 03:47:34 DEBUG "result": [ 03:47:34 DEBUG "metric": { 03:47:34 DEBUG { 03:47:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:47:34 DEBUG "metric": { 03:47:34 DEBUG }, 03:47:34 DEBUG "instance": "10.142.0.107:9100" 03:47:34 DEBUG "value": [ 03:47:34 DEBUG }, 03:47:34 DEBUG 03:47:34 DEBUG "value": [ 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG [...] 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG [...] 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG 03:47:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:47:34 DEBUG 03:47:34 DEBUG }, 03:47:34 DEBUG "instance": "10.142.0.43:9100" 03:47:34 DEBUG "value": [ 03:47:34 DEBUG }, 03:47:34 DEBUG 1713062854, 03:47:34 DEBUG "value": [ 03:47:34 DEBUG "6802870272" 03:47:34 DEBUG 1713062854, 03:47:34 DEBUG ] 03:47:34 DEBUG "0" 03:47:34 DEBUG } 03:47:34 DEBUG ] 03:47:34 DEBUG ] 03:47:34 DEBUG } 03:47:34 DEBUG } 03:47:34 DEBUG ] 03:47:34 DEBUG } 03:47:34 DEBUG } 03:47:34 DEBUG } 03:47:34 INFO 03:47:34 INFO [loop_until]: kubectl --namespace=xlou top pods 03:47:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:47:34 INFO 03:47:34 INFO [loop_until]: kubectl --namespace=xlou top node 03:47:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:47:34 INFO [loop_until]: OK (rc = 0) 03:47:34 DEBUG --- stdout --- 03:47:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 6354m 8942Mi am-7d597cf7d6-8vr8f 5902m 8579Mi am-7d597cf7d6-9wz9h 8127m 8543Mi ds-cts-0 4528m 7198Mi ds-cts-1 5409m 7923Mi ds-cts-2 5327m 8165Mi ds-idrepo-0 6908m 4581Mi ds-idrepo-1 1086m 4293Mi ds-idrepo-2 1026m 4104Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 990Mi idm-b85f9d978-rl76f 4m 987Mi lodemon-66fcb5cd8b-xh4df 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 7892m 7289Mi 03:47:34 DEBUG --- stderr --- 03:47:34 DEBUG 03:47:34 INFO [loop_until]: OK (rc = 0) 03:47:34 DEBUG --- stdout --- 03:47:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 64m 0% 4811Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 5357m 17% 10319Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 6422m 21% 10978Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 10712m 35% 3620Mi 3% gke-xlou-cdm-default-pool-1b0769c8-w15q 1176m 3% 6358Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 4325m 14% 10614Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 62m 0% 2919Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 6221m 20% 10014Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 1114m 3% 5518Mi 4% gke-xlou-cdm-ds-0f601751-j488 4537m 15% 9418Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 60m 0% 2212Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 7906m 26% 10100Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 6813m 22% 6067Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 8210m 51% 10228Mi 17% 03:47:34 DEBUG --- stderr --- 03:47:34 DEBUG 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING No values available for selected timeframe. 03:48:31 WARNING File /tmp/lodemon_data-20240414_033733/node_disk_read_bytes_total.json does not exist. Skipping average value computation... 127.0.0.1 - - [14/Apr/2024 03:48:31] "GET /monitoring/average?start_time=20240414_024637&stop_time=20240414_024630 HTTP/1.1" 200 - 03:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713062913" 03:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062913" 03:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062913" 03:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062913" 03:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062913" 03:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062913" 03:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713062913" 03:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062913" 03:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062913" 03:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062913" 03:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062913" 03:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062913" 03:48:33 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713062913" 03:48:33 INFO [http_cmd]: http status code OK 03:48:33 INFO [http_cmd]: http status code OK 03:48:33 DEBUG --- status code --- 03:48:33 INFO [http_cmd]: http status code OK 03:48:33 INFO [http_cmd]: http status code OK 03:48:33 INFO [http_cmd]: http status code OK 03:48:33 INFO [http_cmd]: http status code OK 03:48:33 INFO [http_cmd]: http status code OK 03:48:33 INFO [http_cmd]: http status code OK 03:48:33 DEBUG --- status code --- 03:48:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:33 DEBUG --- status code --- 03:48:33 DEBUG --- status code --- 03:48:33 DEBUG --- status code --- 03:48:33 DEBUG --- status code --- 03:48:33 DEBUG --- status code --- 03:48:33 DEBUG --- status code --- 03:48:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:33 DEBUG --- http response --- 03:48:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:33 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG --- http response --- 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG { 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG { 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG { 03:48:34 DEBUG { 03:48:34 DEBUG { 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG { 03:48:34 DEBUG { 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "data": { 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:48:34 ERROR [http_cmd]: ERROR 03:48:34 DEBUG "status": "success", 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "data": { 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "data": { 03:48:34 DEBUG "data": { 03:48:34 DEBUG "data": { 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG ----- output ----- 03:48:34 ERROR --- status code --- 03:48:34 DEBUG "data": { 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG "data": { 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "result": [ 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG { 03:48:34 DEBUG { 03:48:34 DEBUG { 03:48:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "result": [ 03:48:34 DEBUG { 03:48:34 DEBUG "result": [ 03:48:34 DEBUG "result": [ 03:48:34 DEBUG "result": [ 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "status": "success", 03:48:34 ERROR --- http response --- 03:48:34 DEBUG "result": [ 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG "result": [ 03:48:34 DEBUG { 03:48:34 DEBUG "metric": { 03:48:34 DEBUG { 03:48:34 DEBUG { 03:48:34 DEBUG { 03:48:34 DEBUG "data": { 03:48:34 DEBUG "data": { 03:48:34 DEBUG "data": { 03:48:34 DEBUG { 03:48:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:48:34 DEBUG { 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (125 lines): 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "pod": "ds-idrepo-1" 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "metric": { 03:48:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:48:34 DEBUG "metric": { 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:48:34 DEBUG }, 03:48:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:48:34 DEBUG "instance": "10.142.0.107:9100" 03:48:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:48:34 DEBUG "result": [ 03:48:34 DEBUG "result": [ 03:48:34 DEBUG "result": [ 03:48:34 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:48:34 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:48:34 DEBUG { 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG }, 03:48:34 DEBUG }, 03:48:34 DEBUG }, 03:48:34 DEBUG { 03:48:34 DEBUG { 03:48:34 DEBUG { 03:48:34 DEBUG }, 03:48:34 DEBUG }, 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "data": { 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "pod": "ds-idrepo-1" 03:48:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:48:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG }, 03:48:34 DEBUG }, 03:48:34 DEBUG }, 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "result": [ 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG { 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG [...] 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG [...] 03:48:34 DEBUG "metric": { 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG }, 03:48:34 DEBUG 03:48:34 DEBUG "pod": "ds-idrepo-0" 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG [...] 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:48:34 DEBUG }, 03:48:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:48:34 DEBUG "instance": "10.142.0.43:9100" 03:48:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "pod": "idm-b85f9d978-j5pbm" 03:48:34 DEBUG "pod": "idm-b85f9d978-j5pbm" 03:48:34 DEBUG 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG }, 03:48:34 DEBUG }, 03:48:34 DEBUG }, 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG }, 03:48:34 DEBUG }, 03:48:34 DEBUG 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 1713062913, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG 1713062913, 03:48:34 DEBUG "0" 03:48:34 DEBUG 1713062913, 03:48:34 DEBUG 1713062913, 03:48:34 DEBUG 1713062913, 03:48:34 DEBUG "pod": "ds-idrepo-0" 03:48:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:48:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:48:34 DEBUG 1713062913, 03:48:34 DEBUG 1713062913, 03:48:34 DEBUG [...] 03:48:34 DEBUG "0.015" 03:48:34 DEBUG ] 03:48:34 DEBUG "232525" 03:48:34 DEBUG "12672748544" 03:48:34 DEBUG "8.034650993528597" 03:48:34 DEBUG }, 03:48:34 DEBUG }, 03:48:34 DEBUG }, 03:48:34 DEBUG "0" 03:48:34 DEBUG "9044.708274683975" 03:48:34 DEBUG 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG ] 03:48:34 DEBUG ] 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "value": [ 03:48:34 DEBUG ] 03:48:34 DEBUG ] 03:48:34 DEBUG 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG 1713062913, 03:48:34 DEBUG 1713062913, 03:48:34 DEBUG 1713062913, 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG ] 03:48:34 DEBUG ] 03:48:34 DEBUG "0" 03:48:34 DEBUG "0" 03:48:34 DEBUG "855.7333333333333" 03:48:34 DEBUG ] 03:48:34 DEBUG ] 03:48:34 DEBUG "pod": "idm-b85f9d978-j5pbm" 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG ] 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG 1713062913, 03:48:34 DEBUG ] 03:48:34 DEBUG ] 03:48:34 DEBUG ] 03:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062914" 03:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062914" 03:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062914" 03:48:34 DEBUG "0" 03:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062914" 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062914" 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG --- status code --- 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG { 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG { 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG { 03:48:34 DEBUG "status": "success", 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG "data": { 03:48:34 DEBUG "status": "success", 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG "data": { 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "data": { 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG "result": [ 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG "result": [ 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG { 03:48:34 DEBUG "result": [ 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG { 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:48:34 DEBUG "metric": { 03:48:34 DEBUG { 03:48:34 DEBUG "metric": { 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG "pod": "ds-idrepo-1" 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "pod": "ds-idrepo-1" 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG { 03:48:34 DEBUG }, 03:48:34 DEBUG "pod": "ds-idrepo-1" 03:48:34 DEBUG }, 03:48:34 DEBUG { 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "value": [ 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "data": { 03:48:34 DEBUG 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG "data": { 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "result": [ 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "result": [ 03:48:34 DEBUG { 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG { 03:48:34 DEBUG "metric": { 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG "metric": { 03:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062914" 03:48:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "instance": "10.142.0.107:9100" 03:48:34 DEBUG }, 03:48:34 DEBUG 03:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062914" 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "pod": "ds-idrepo-0" 03:48:34 DEBUG 03:48:34 DEBUG "pod": "ds-cts-1" 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG }, 03:48:34 DEBUG "pod": "ds-idrepo-0" 03:48:34 DEBUG }, 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "value": [ 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 1713062914, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 1713062914, 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG "0" 03:48:34 DEBUG 1713062914, 03:48:34 DEBUG "0.00017925" 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG ] 03:48:34 DEBUG "0" 03:48:34 DEBUG ] 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG 03:48:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG "instance": "10.142.0.43:9100" 03:48:34 DEBUG }, 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062914" 03:48:34 DEBUG } 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG } 03:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062914" 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 1713062914, 03:48:34 DEBUG 1713062914, 03:48:34 DEBUG "794.6" 03:48:34 DEBUG "2947.4666666666667" 03:48:34 DEBUG ] 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713062914" 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG { 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "data": { 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "result": [ 03:48:34 DEBUG { 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "pod": "ds-idrepo-1" 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "pod": "ds-idrepo-0" 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 1713062914, 03:48:34 DEBUG "0" 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG { 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "data": { 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "result": [ 03:48:34 DEBUG { 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "pod": "idm-b85f9d978-j5pbm" 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 1713062914, 03:48:34 DEBUG "0.0012813105255666864" 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG { 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG "data": { 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "result": [ 03:48:34 DEBUG { 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "instance": "10.142.0.107:9100" 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "instance": "10.142.0.43:9100" 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 1713062914, 03:48:34 DEBUG "0" 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG { 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "data": { 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "result": [ 03:48:34 DEBUG { 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 1713062914, 03:48:34 DEBUG "794.5666666666667" 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG { 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "data": { 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "result": [ 03:48:34 DEBUG { 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "pod": "ds-idrepo-1" 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "pod": "ds-cts-1" 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 1713062914, 03:48:34 DEBUG "0" 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062914" 03:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062914" 03:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062914" 03:48:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713062914" 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG { 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "data": { 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "result": [ 03:48:34 DEBUG { 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:48:34 DEBUG }, 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 1713062914, 03:48:34 DEBUG "794.5666666666666" 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG --- http response --- 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 INFO [http_cmd]: http status code OK 03:48:34 DEBUG { 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG --- status code --- 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:48:34 DEBUG "data": { 03:48:34 DEBUG --- http response --- 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG { 03:48:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:48:34 DEBUG "result": [ 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG ----- output ----- 03:48:34 DEBUG { 03:48:34 DEBUG "data": { 03:48:34 DEBUG { 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "status": "success", 03:48:34 DEBUG "instance": "10.142.0.107:9100" 03:48:34 DEBUG "result": [ 03:48:34 DEBUG "data": { 03:48:34 DEBUG }, 03:48:34 DEBUG { 03:48:34 DEBUG "resultType": "vector", 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "metric": { 03:48:34 DEBUG "result": [ 03:48:34 DEBUG 03:48:34 DEBUG "instance": "10.142.0.107:9100" 03:48:34 DEBUG { 03:48:34 DEBUG 03:48:34 DEBUG }, 03:48:34 DEBUG "metric": { 03:48:34 DEBUG 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG }, 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG 03:48:34 DEBUG "instance": "10.142.0.43:9100" 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG }, 03:48:34 DEBUG 03:48:34 DEBUG [...] 03:48:34 DEBUG "value": [ 03:48:34 DEBUG 03:48:34 DEBUG 03:48:34 DEBUG 1713062914, 03:48:34 DEBUG "instance": "10.142.0.43:9100" 03:48:34 DEBUG 03:48:34 DEBUG "5250.900000000001" 03:48:34 DEBUG }, 03:48:34 DEBUG 03:48:34 DEBUG ] 03:48:34 DEBUG "value": [ 03:48:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:48:34 DEBUG } 03:48:34 DEBUG 1713062914, 03:48:34 DEBUG }, 03:48:34 DEBUG ] 03:48:34 DEBUG "0.00036666666666675953" 03:48:34 DEBUG "value": [ 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG 1713062914, 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG "8466980864" 03:48:34 DEBUG ] 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 DEBUG ] 03:48:34 DEBUG } 03:48:34 DEBUG } 03:48:34 INFO 03:48:34 INFO [loop_until]: kubectl --namespace=xlou top pods 03:48:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:48:34 INFO [loop_until]: OK (rc = 0) 03:48:34 DEBUG --- stdout --- 03:48:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 3760m 8961Mi am-7d597cf7d6-8vr8f 3919m 8580Mi am-7d597cf7d6-9wz9h 5259m 8537Mi ds-cts-0 3077m 7193Mi ds-cts-1 3123m 7948Mi ds-cts-2 3322m 8150Mi ds-idrepo-0 4855m 4580Mi ds-idrepo-1 675m 4293Mi ds-idrepo-2 568m 4104Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 4m 990Mi idm-b85f9d978-rl76f 4m 987Mi lodemon-66fcb5cd8b-xh4df 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 6657m 16Mi 03:48:34 DEBUG --- stderr --- 03:48:34 DEBUG 03:48:34 INFO 03:48:34 INFO [loop_until]: kubectl --namespace=xlou top node 03:48:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:48:34 INFO [loop_until]: OK (rc = 0) 03:48:34 DEBUG --- stdout --- 03:48:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 63m 0% 4810Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3604m 12% 10342Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 4094m 13% 11029Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 6072m 20% 3090Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 749m 2% 6361Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 4336m 14% 10629Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 63m 0% 2922Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 3764m 12% 10029Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 680m 2% 5515Mi 4% gke-xlou-cdm-ds-0f601751-j488 4443m 14% 9448Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 62m 0% 2215Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 5999m 20% 10108Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 3548m 11% 6055Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 4127m 25% 3930Mi 6% 03:48:34 DEBUG --- stderr --- 03:48:34 DEBUG 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_read_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%29%29by%28instance%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_entry_count%7Bbackend%3D~%27amCts%27%2Cpod%3D~%27ctsstore.%2A%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28node_disk_written_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_writes_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_cpu_usage_seconds_total%3Asum_irate%29by%28node%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_reaper_deletion_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 ERROR [http_cmd]: ERROR 03:49:34 DEBUG --- status code --- 03:49:34 ERROR --- status code --- 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 ERROR http status code is 400 (expected [200, 401, 403, 404]) 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amCts%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 DEBUG --- http response --- 03:49:34 ERROR --- http response --- 03:49:34 DEBUG { "status": "success", "data": { "resultType": "vector", "result": [] } } 03:49:34 DEBUG { "status": "error", "errorType": "bad_data", "error": "invalid parameter \"query\": 1:10: parse error: expected type range vector in call to function \"rate\", got instant vector" } 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 ERROR Query: sum(rate(node_disk_read_bytes_total{job='node-exporter',device=~'nvme.+|rbd.+|sd.+|vd.+|xvd.+|dasd.+'}))by(instance) failed with: [http_cmd] request returns wrong status code 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG { 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG "data": { 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG "result": [ 03:49:34 DEBUG { 03:49:34 DEBUG "metric": { 03:49:34 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG [...] 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "1241999.1929440247" 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG { 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG "data": { 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG "result": [ 03:49:34 DEBUG { 03:49:34 DEBUG "metric": { 03:49:34 DEBUG "instance": "10.142.0.107:9100" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG [...] 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "instance": "10.142.0.43:9100" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "12673412096" 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG { 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG "data": { 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG "result": [ 03:49:34 DEBUG { 03:49:34 DEBUG "metric": { 03:49:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG [...] 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "0.030591182265966285" 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG { 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG "data": { 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG "result": [ 03:49:34 DEBUG { 03:49:34 DEBUG "metric": { 03:49:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG [...] 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "1314.3" 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG { 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG "data": { 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG "result": [ 03:49:34 DEBUG { 03:49:34 DEBUG "metric": { 03:49:34 DEBUG "pod": "ds-idrepo-1" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG [...] 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "pod": "ds-idrepo-0" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "0" 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27cfgStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_replayed_updates_conflicts_resolved%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_search_seconds_total%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_memory_working_set_bytes%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28avg_over_time%28am_cts_reaper_cache_size%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_oauth2_grant_count%7Bnamespace%3D%27xlou%27%2Cgrant_type%3D~%27authorization-code%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_fs_reads_bytes_total%7Bnamespace%3D%27xlou%27%2Cjob%3D%27kubelet%27%2Cmetrics_path%3D%27/metrics/cadvisor%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG { 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG { 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG { 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG "data": { 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG "data": { 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG "data": { 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:49:34 DEBUG "result": [ 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG "result": [ 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG { 03:49:34 DEBUG "result": [ 03:49:34 DEBUG { 03:49:34 DEBUG { 03:49:34 DEBUG { 03:49:34 DEBUG "metric": { 03:49:34 DEBUG { 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG "metric": { 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG "pod": "ds-idrepo-1" 03:49:34 DEBUG "metric": { 03:49:34 DEBUG "data": { 03:49:34 DEBUG "pod": "ds-idrepo-1" 03:49:34 DEBUG "data": { 03:49:34 DEBUG }, 03:49:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG }, 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG "value": [ 03:49:34 DEBUG }, 03:49:34 DEBUG "result": [ 03:49:34 DEBUG "value": [ 03:49:34 DEBUG "result": [ 03:49:34 DEBUG 03:49:34 DEBUG "value": [ 03:49:34 DEBUG { 03:49:34 DEBUG 03:49:34 DEBUG { 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "metric": { 03:49:34 DEBUG 03:49:34 DEBUG "metric": { 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:49:34 DEBUG 03:49:34 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:49:34 DEBUG [...] 03:49:34 DEBUG 03:49:34 DEBUG }, 03:49:34 DEBUG [...] 03:49:34 DEBUG }, 03:49:34 DEBUG 03:49:34 DEBUG [...] 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "pod": "ds-cts-1" 03:49:34 DEBUG 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG 03:49:34 DEBUG "pod": "ds-idrepo-0" 03:49:34 DEBUG 03:49:34 DEBUG }, 03:49:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG [...] 03:49:34 DEBUG }, 03:49:34 DEBUG [...] 03:49:34 DEBUG "value": [ 03:49:34 DEBUG }, 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG 03:49:34 DEBUG "0" 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG "0" 03:49:34 DEBUG 03:49:34 DEBUG ] 03:49:34 DEBUG "0.015" 03:49:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:49:34 DEBUG { 03:49:34 DEBUG ] 03:49:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG }, 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG } 03:49:34 DEBUG }, 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG "value": [ 03:49:34 DEBUG "data": { 03:49:34 DEBUG ] 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG "value": [ 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG } 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 DEBUG "156579.5" 03:49:34 DEBUG "result": [ 03:49:34 DEBUG } 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG "256057023.3469764" 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG { 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG "metric": { 03:49:34 DEBUG } 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:49:34 DEBUG ] 03:49:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:49:34 DEBUG ] 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG } 03:49:34 DEBUG }, 03:49:34 DEBUG } 03:49:34 DEBUG { 03:49:34 DEBUG } 03:49:34 DEBUG "value": [ 03:49:34 DEBUG } 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG 03:49:34 DEBUG "data": { 03:49:34 DEBUG 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG 03:49:34 DEBUG "result": [ 03:49:34 DEBUG [...] 03:49:34 DEBUG { 03:49:34 DEBUG 03:49:34 DEBUG "metric": { 03:49:34 DEBUG 03:49:34 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:49:34 DEBUG 03:49:34 DEBUG }, 03:49:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:49:34 DEBUG "value": [ 03:49:34 DEBUG }, 03:49:34 DEBUG 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG 03:49:34 DEBUG "0" 03:49:34 DEBUG [...] 03:49:34 DEBUG ] 03:49:34 DEBUG 03:49:34 DEBUG } 03:49:34 DEBUG 03:49:34 DEBUG ] 03:49:34 DEBUG 03:49:34 DEBUG } 03:49:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:49:34 DEBUG } 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27amIdentityStore%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "0" 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_replication_replica_remote_replicas_receive_delay_seconds+%7Bnamespace%3D%27xlou%27%2Cdomain_name%3D%27ou%3Dtokens%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_ttl_entries_deleted_count%7Bnamespace%3D%27xlou%27%2Cbackend%3D~%27amCts%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_cts_task_count%7Btoken_type%3D%27session%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_receive_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28ds_backend_db_cache_misses_internal_nodes%7Bbackend%3D%27idmRepo%27%2Cnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28container_cpu_usage_seconds_total%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (62 lines): 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (53 lines): 03:49:34 DEBUG ----- output ----- 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG { 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG { 03:49:34 DEBUG { 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG "data": { 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG "data": { 03:49:34 DEBUG "data": { 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:49:34 DEBUG ----- output ----- 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28irate%28am_session_count%7Bsession_type%3D~%27authentication-.%2A%27%2Coperation%3D%27create%27%2Cnamespace%3D%27xlou%27%2Coutcome%3D%27success%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 DEBUG "resultType": "vector", 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG "result": [ 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_cpu_seconds_total%7Bmode%3D%27iowait%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062974" 03:49:34 DEBUG { 03:49:34 DEBUG "result": [ 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG "result": [ 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG { 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG { 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG { 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG "metric": { 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG "data": { 03:49:34 DEBUG "metric": { 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG "metric": { 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG "pod": "ds-idrepo-1" 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG "pod": "ds-idrepo-1" 03:49:34 DEBUG "pod": "ds-idrepo-1" 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG }, 03:49:34 DEBUG "result": [ 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:49:34 DEBUG }, 03:49:34 DEBUG }, 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG "value": [ 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (143 lines): 03:49:34 DEBUG { 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG "value": [ 03:49:34 DEBUG "value": [ 03:49:34 DEBUG { 03:49:34 DEBUG 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG "metric": { 03:49:34 DEBUG { 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG 03:49:34 DEBUG { 03:49:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "data": { 03:49:34 DEBUG 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG }, 03:49:34 DEBUG "data": { 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG [...] 03:49:34 DEBUG "data": { 03:49:34 DEBUG "value": [ 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG [...] 03:49:34 DEBUG [...] 03:49:34 DEBUG "result": [ 03:49:34 DEBUG 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG 03:49:34 DEBUG "result": [ 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG { 03:49:34 DEBUG 03:49:34 DEBUG "result": [ 03:49:34 DEBUG 03:49:34 DEBUG { 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "metric": { 03:49:34 DEBUG 03:49:34 DEBUG { 03:49:34 DEBUG 03:49:34 DEBUG "metric": { 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "pod": "ds-idrepo-1" 03:49:34 DEBUG "pod": "ds-idrepo-0" 03:49:34 DEBUG "metric": { 03:49:34 DEBUG [...] 03:49:34 DEBUG "instance": "10.142.0.107:9100" 03:49:34 DEBUG "pod": "ds-idrepo-0" 03:49:34 DEBUG "pod": "ds-cts-1" 03:49:34 DEBUG }, 03:49:34 DEBUG }, 03:49:34 DEBUG "pod": "admin-ui-c4c7bf584-mbzlq" 03:49:34 DEBUG 03:49:34 DEBUG }, 03:49:34 DEBUG }, 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG "value": [ 03:49:34 DEBUG }, 03:49:34 DEBUG 03:49:34 DEBUG "value": [ 03:49:34 DEBUG "value": [ 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG 03:49:34 DEBUG "0" 03:49:34 DEBUG 03:49:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:49:34 DEBUG 03:49:34 DEBUG "0" 03:49:34 DEBUG "0.0001666666666666667" 03:49:34 DEBUG 03:49:34 DEBUG ] 03:49:34 DEBUG 03:49:34 DEBUG }, 03:49:34 DEBUG 03:49:34 DEBUG ] 03:49:34 DEBUG ] 03:49:34 DEBUG [...] 03:49:34 DEBUG } 03:49:34 DEBUG 03:49:34 DEBUG "value": [ 03:49:34 DEBUG [...] 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG [...] 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG 03:49:34 DEBUG ] 03:49:34 DEBUG ] 03:49:34 DEBUG ] 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "0" 03:49:34 DEBUG 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 DEBUG "pod": "ds-idrepo-0" 03:49:34 DEBUG 03:49:34 DEBUG ] 03:49:34 DEBUG 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 DEBUG }, 03:49:34 DEBUG 03:49:34 DEBUG } 03:49:34 DEBUG "instance": "10.142.0.43:9100" 03:49:34 DEBUG "value": [ 03:49:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:49:34 DEBUG ] 03:49:34 DEBUG }, 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG }, 03:49:34 DEBUG } 03:49:34 DEBUG "value": [ 03:49:34 DEBUG "0" 03:49:34 DEBUG "value": [ 03:49:34 DEBUG } 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG ] 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "6976.333333333333" 03:49:34 DEBUG } 03:49:34 DEBUG "1.5055375857496247" 03:49:34 DEBUG ] 03:49:34 DEBUG ] 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG } 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG { 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG "data": { 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG "result": [ 03:49:34 DEBUG { 03:49:34 DEBUG "metric": { 03:49:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG [...] 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "0" 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG { 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG "data": { 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG "result": [ 03:49:34 DEBUG { 03:49:34 DEBUG "metric": { 03:49:34 DEBUG "instance": "10.142.0.107:9100" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG [...] 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "instance": "10.142.0.43:9100" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "0" 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_disk_io_time_seconds_total%7Bjob%3D%27node-exporter%27%2Cdevice%3D~%27nvme.%2B%7Crbd.%2B%7Csd.%2B%7Cvd.%2B%7Cxvd.%2B%7Cdasd.%2B%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28am_authentication_count%7Bnamespace%3D%27xlou%27%7D%5B60s%5D%29%29by%28pod%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28rate%28node_network_transmit_bytes_total%7Bjob%3D%27node-exporter%27%2Cdevice%21%3D%27lo%27%7D%5B60s%5D%29%29by%28instance%29&time=1713062974" 03:49:34 INFO [http_cmd]: curl --insecure -L --request GET "http://prometheus-operator-kube-p-prometheus.monitoring.svc.cluster.local:9090/api/v1/query?query=sum%28node_namespace_pod_container%3Acontainer_memory_working_set_bytes%29by%28node%29&time=1713062974" 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG --- http response --- 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (35 lines): 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG { 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:49:34 DEBUG "data": { 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG { 03:49:34 DEBUG "result": [ 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG { 03:49:34 DEBUG "data": { 03:49:34 DEBUG "metric": { 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG "pod": "am-7d597cf7d6-8ctft" 03:49:34 DEBUG "result": [ 03:49:34 DEBUG }, 03:49:34 DEBUG { 03:49:34 DEBUG "value": [ 03:49:34 DEBUG "metric": { 03:49:34 DEBUG 03:49:34 DEBUG "instance": "10.142.0.107:9100" 03:49:34 DEBUG 03:49:34 DEBUG }, 03:49:34 DEBUG 03:49:34 DEBUG "value": [ 03:49:34 DEBUG [...] 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG [...] 03:49:34 DEBUG "pod": "am-7d597cf7d6-9wz9h" 03:49:34 DEBUG 03:49:34 DEBUG }, 03:49:34 DEBUG 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "instance": "10.142.0.43:9100" 03:49:34 DEBUG "0" 03:49:34 DEBUG }, 03:49:34 DEBUG ] 03:49:34 DEBUG "value": [ 03:49:34 DEBUG } 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG ] 03:49:34 DEBUG "0.00030000000000001136" 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG { 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG "data": { 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG "result": [ 03:49:34 DEBUG { 03:49:34 DEBUG "metric": { 03:49:34 DEBUG "instance": "10.142.0.107:9100" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG [...] 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "instance": "10.142.0.43:9100" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "213671.53333333333" 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 INFO [http_cmd]: http status code OK 03:49:34 DEBUG --- status code --- 03:49:34 DEBUG http status code is 200 (expected [200, 401, 403, 404]) 03:49:34 DEBUG --- http response --- 03:49:34 DEBUG [print_head_tail]: Print head (10 lines) and tail (10 lines) of input string (134 lines): 03:49:34 DEBUG ----- output ----- 03:49:34 DEBUG { 03:49:34 DEBUG "status": "success", 03:49:34 DEBUG "data": { 03:49:34 DEBUG "resultType": "vector", 03:49:34 DEBUG "result": [ 03:49:34 DEBUG { 03:49:34 DEBUG "metric": { 03:49:34 DEBUG "node": "gke-xlou-cdm-ds-0f601751-mc5d" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG [...] 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG 03:49:34 DEBUG "node": "gke-xlou-cdm-frontend-a8771548-i82n" 03:49:34 DEBUG }, 03:49:34 DEBUG "value": [ 03:49:34 DEBUG 1713062974, 03:49:34 DEBUG "113500160" 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG ] 03:49:34 DEBUG } 03:49:34 DEBUG } 03:49:34 INFO 03:49:34 INFO [loop_until]: kubectl --namespace=xlou top pods 03:49:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:49:34 INFO 03:49:34 INFO [loop_until]: kubectl --namespace=xlou top node 03:49:34 INFO [loop_until]: (max_time=180, interval=5, expected_rc=[0] 03:49:34 INFO [loop_until]: OK (rc = 0) 03:49:34 DEBUG --- stdout --- 03:49:34 DEBUG NAME CPU(cores) MEMORY(bytes) admin-ui-c4c7bf584-mbzlq 1m 8Mi am-7d597cf7d6-8ctft 1193m 8929Mi am-7d597cf7d6-8vr8f 1110m 8556Mi am-7d597cf7d6-9wz9h 1471m 8471Mi ds-cts-0 3000m 7183Mi ds-cts-1 3031m 7949Mi ds-cts-2 3106m 8166Mi ds-idrepo-0 923m 4581Mi ds-idrepo-1 254m 4292Mi ds-idrepo-2 290m 4105Mi end-user-ui-58878d7668-zg4h2 1m 5Mi idm-b85f9d978-j5pbm 13m 991Mi idm-b85f9d978-rl76f 10m 988Mi lodemon-66fcb5cd8b-xh4df 2m 63Mi login-ui-5fdf49fc66-2wnx7 1m 4Mi overseer-0-5584f6c4db-bcn5q 849m 127Mi 03:49:34 DEBUG --- stderr --- 03:49:34 DEBUG 03:49:34 INFO [loop_until]: OK (rc = 0) 03:49:34 DEBUG --- stdout --- 03:49:34 DEBUG NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% gke-xlou-cdm-default-pool-1b0769c8-7vmj 72m 0% 4814Mi 4% gke-xlou-cdm-default-pool-1b0769c8-mpfw 3125m 10% 10319Mi 9% gke-xlou-cdm-default-pool-1b0769c8-nhlf 1236m 4% 11023Mi 9% gke-xlou-cdm-default-pool-1b0769c8-qkns 96m 0% 3058Mi 2% gke-xlou-cdm-default-pool-1b0769c8-w15q 335m 1% 6315Mi 5% gke-xlou-cdm-default-pool-1b0769c8-wpwl 3146m 10% 10622Mi 9% gke-xlou-cdm-default-pool-1b0769c8-xnmp 78m 0% 2918Mi 2% gke-xlou-cdm-ds-0f601751-cf7b 1279m 4% 10021Mi 8% gke-xlou-cdm-ds-0f601751-hmm7 341m 1% 5471Mi 4% gke-xlou-cdm-ds-0f601751-j488 3015m 10% 9431Mi 8% gke-xlou-cdm-ds-0f601751-mc5d 70m 0% 2219Mi 1% gke-xlou-cdm-ds-0f601751-nj6q 1647m 5% 10077Mi 9% gke-xlou-cdm-ds-0f601751-nmxm 1098m 3% 6074Mi 5% gke-xlou-cdm-frontend-a8771548-i82n 921m 5% 3946Mi 6% 03:49:34 DEBUG --- stderr --- 03:49:34 DEBUG 03:50:13 INFO Finished: True 03:50:13 INFO Waiting for threads to register finish flag 03:50:34 INFO Done. Have a nice day! :) 127.0.0.1 - - [14/Apr/2024 03:50:34] "GET /monitoring/stop HTTP/1.1" 200 - 03:50:37 WARNING File /tmp/lodemon_data-20240414_033733/node_disk_read_bytes_total.json does not exist. Skipping graph generation... WARNING:bokeh.core.validation.check:W-1000 (MISSING_RENDERERS): Plot has no renderers: figure(id='p6511', ...) 127.0.0.1 - - [14/Apr/2024 03:50:41] "GET /monitoring/process HTTP/1.1" 200 -